Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 107 | 61.68% (off track) | -33.32pp | +19.32pp | 84.11% (90/107) | -3.86pp | 73.33% (66/90) | +21.20pp |
| INT | 95% - All E2E job runs | 22 | 72.73% (off track) | -22.27pp | +1.30pp | n/a | n/a | 100.00% (16/16) | +3.23pp |
| STG | 95% - All E2E job runs | 20 | 85.00% (off track) | -10.00pp | +33.48pp | n/a | n/a | 89.47% (17/19) | +18.64pp |
| PROD | 95% - All E2E job runs | 65 | 73.85% (off track) | -21.15pp | +41.89pp | n/a | n/a | 73.85% (48/65) | +40.51pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 10 | 3.98%10 of 251 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsNov 7: 0 · Nov 8: 1 · Nov 9: 1 · Nov 10: 2 · Nov 11: 1 · Nov 12: 2 · Nov 13: 3 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 32 time(s) (14 unknown, 18 known) Firing 1: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster", pod="kube-apiserver-578576bffb-f6whx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster/kube-apiserver-578576bffb-f6whx has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:41:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="router-c76569fc9-6cfvc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/router-c76569fc9-6cfvc has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:41:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpes06vummjujuaun3cf6766c5d4-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpes06vummjujuaun3cf6766c5d4-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-11-13T22:41:33Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvd6vuver3peg6br0ldulhn5em1d2-cluster", pod="kube-state-metrics-hcp-867779fc97-5kqgw", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvd6vuver3peg6br0ldulhn5em1d2-cluster/kube-state-metrics-hcp-867779fc97-5kqgw has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:49:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster", pod="openshift-apiserver-548798444f-fkvbc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster/openshift-apiserver-548798444f-fkvbc has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster", pod="openshift-apiserver-7db68f8b9c-ztk8f", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster/openshift-apiserver-7db68f8b9c-ztk8f has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2k00np1fd7kopst9snbjsh5ghmkl25-cluster", pod="kube-state-metrics-hcp-867779fc97-4rb5d", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2k00np1fd7kopst9snbjsh5ghmkl25-cluster/kube-state-metrics-hcp-867779fc97-4rb5d has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster", pod="kube-controller-manager-7b9955c77b-gt9w8", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster/kube-controller-manager-7b9955c77b-gt9w8 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2k00tuj5acf49s1c11mi5h9ldau5ps-cluster", pod="openshift-apiserver-69b9c969cf-2bfxx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2k00tuj5acf49s1c11mi5h9ldau5ps-cluster/openshift-apiserver-69b9c969cf-2bfxx has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="kube-state-metrics-hcp-867779fc97-nnqft", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/kube-state-metrics-hcp-867779fc97-nnqft has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-11-13T22:46:32Z Ended: 2026-11-13T22:52:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juvn997fddu0ckuf17ouet8eoim9t-cluster", pod="openshift-apiserver-f95d96f84-p58r8", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juvn997fddu0ckuf17ouet8eoim9t-cluster/openshift-apiserver-f95d96f84-p58r8 has been in a non-ready state for longer than 5 minutes. alert fired 14 time(s) (1 unknown, 13 known) Firing 1: State: Resolved Started: 2026-11-13T13:12:50Z Ended: 2026-11-13T13:28:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5993600-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster", pod="etcd-1", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-13T09:56:23Z Ended: 2026-11-13T10:04:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2098432-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u28rdtkqoukt96fddb92fogemvp70gv-cluster", pod="openshift-route-controller-manager-8589b467c7-s9rts", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u28rdtkqoukt96fddb92fogemvp70gv-cluster/openshift-route-controller-manager-8589b467c7-s9rts has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGridfull failure pattern: ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | provision | 9 | 3.59%9 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 3 · Nov 10: 3 · Nov 11: 3 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-11T14:26:38.768Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T14:26:40.146Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:26:42.295Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:26:46.642Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:26:55.498Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:27:12.634Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:27:45.036Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:27:45.036Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T14:27:46.265Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9432960%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fa5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b
time=2026-11-11T14:43:19.581Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b/operationStatuses/08584071996199689084\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 64f93b66-743f-4a72-adcd-148ae5d247c0\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 64f93b66-743f-4a72-adcd-148ae5d247c0\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9432960/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j9432960\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-11T00:47:48.881Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T00:47:50.316Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:47:52.553Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:47:56.856Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:05.395Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:22.613Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:57.747Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:57.748Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T00:48:58.504Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3478912%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908
time=2026-11-11T01:04:33.988Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908/operationStatuses/08584072487474526673\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2e23547e-99e6-4dbe-b935-a08d562df87a\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2e23547e-99e6-4dbe-b935-a08d562df87a\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3478912/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j3478912\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-11T00:47:18.513Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T00:47:20.111Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:22.298Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:26.447Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:34.929Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:51.119Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:48:23.772Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:48:23.772Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T00:48:24.640Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2734720%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81
time=2026-11-11T01:03:58.863Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81/operationStatuses/08584072487812695254\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: bc3822da-78e4-4890-b47c-d63a893c752f\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: bc3822da-78e4-4890-b47c-d63a893c752f\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2734720/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j2734720\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 7 | 2.79%7 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 1 · Nov 11: 0 · Nov 12: 4 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-5-0-fvh62c"
Unexpected error:
<*errors.joinError | 0xe9c5185e558>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-fvh62c-qp98f-4xr6k": nodes "cluster-np-fvh62c-qp98f-4xr6k" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xe9c50b2e640>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e620>{
msg: "failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e600>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xe9c518f6f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-fvh62c-qp98f-4xr6k",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-5-0-fvh62c"
Unexpected error:
<*errors.joinError | 0xe9c5185e558>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-fvh62c-qp98f-4xr6k": nodes "cluster-np-fvh62c-qp98f-4xr6k" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xe9c50b2e640>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e620>{
msg: "failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e600>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xe9c518f6f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-fvh62c-qp98f-4xr6k",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-xcrqj2"
Unexpected error:
<*errors.joinError | 0x1ff6e5be7bd8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xcrqj2-s4s2f-4fg9q": nodes "cluster-np-xcrqj2-s4s2f-4fg9q" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1ff6e6292100>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920e0>{
msg: "failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920c0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1ff6e62961e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xcrqj2-s4s2f-4fg9q",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-xcrqj2"
Unexpected error:
<*errors.joinError | 0x1ff6e5be7bd8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xcrqj2-s4s2f-4fg9q": nodes "cluster-np-xcrqj2-s4s2f-4fg9q" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1ff6e6292100>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920e0>{
msg: "failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920c0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1ff6e62961e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xcrqj2-s4s2f-4fg9q",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7z7qh5"
Unexpected error:
<*errors.joinError | 0x3ff8850ea50>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-7z7qh5-5kzdr-5p27s": nodes "cluster-np-7z7qh5-5kzdr-5p27s" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x3ff87627d80>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627d00>{
msg: "failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627cc0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x3ff884aeb40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-7z7qh5-5kzdr-5p27s",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7z7qh5"
Unexpected error:
<*errors.joinError | 0x3ff8850ea50>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-7z7qh5-5kzdr-5p27s": nodes "cluster-np-7z7qh5-5kzdr-5p27s" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x3ff87627d80>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627d00>{
msg: "failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627cc0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x3ff884aeb40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-7z7qh5-5kzdr-5p27s",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerRetryHotLoop firedfull failure pattern: alert [svc] BackendControllerRetryHotLoop fired Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | alert | 5 | 1.99%5 of 251 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 2 · Nov 11: 2 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Resolved Started: 2026-11-12T15:43:07Z Ended: 2026-11-12T15:54:07Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 2: State: Resolved Started: 2026-11-12T16:08:07Z Ended: 2026-11-12T16:16:07Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 3: State: Resolved Started: 2026-11-12T16:37:06Z Ended: 2026-11-12T16:50:06Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 4: State: Fired (not resolved) Started: 2026-11-12T17:37:08Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-11-11T21:59:13Z Ended: 2026-11-11T22:27:13Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j3946112-svc", component="backend", name="placement", region="canadacentral", severity="warning" Description: Backend controller workqueue placement has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 2: State: Resolved Started: 2026-11-11T22:08:15Z Ended: 2026-11-11T22:24:13Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j3946112-svc", component="backend", name="identityroleassignments", region="canadacentral", severity="warning" Description: Backend controller workqueue identityroleassignments has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 3: State: Fired (not resolved) Started: 2026-11-11T22:31:14Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j3946112-svc", component="backend", name="ensuremanagedresourcegroup", region="canadacentral", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-11-11T20:14:30Z Ended: 2026-11-11T20:21:35Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j0146816-svc", component="backend", name="deleteorphanedcosmosresources", region="westus3", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 2: State: Fired (not resolved) Started: 2026-11-11T20:46:29Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j0146816-svc", component="backend", name="deleteorphanedcosmosresources", region="westus3", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...full failure pattern: ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDBSignal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | provision | 4 | 1.59%4 of 251 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-12T02:15:15.432Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-12T02:15:18.214Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:20.478Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:24.642Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:33.477Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:50.325Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:16:22.712Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:16:22.712Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-12T02:16:23.692Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4812544%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6
time=2026-11-12T02:22:55.890Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6/operationStatuses/08584071571022698221\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8e807b60-5023-4afe-b65a-da1bd577fad4\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8e807b60-5023-4afe-b65a-da1bd577fad4\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4812544/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j4812544\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 49b4ee9d-91eb-463e-be2a-f9a6a2f0c36a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: b3cae6cf-1e2a-46fe-90b3-6a504a79fd6f, Request URI: /apps/457b0b96-1859-441b-9897-a8af2244ff42/services/0882f1b4-9e5e-44a9-a23d-5aff02eaa165/partitions/d4426837-d7ca-44b7-a111-280431846736/replicas/134337933848286181p, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-12T02:22:03.6687109Z, RequestEndTime: 2026-11-12T02:22:09.6687149Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:15.3229628Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.585,\\\\\\\"memory\\\\\\\":23308184.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1208,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":119},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:25.3386251Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.854,\\\\\\\"memory\\\\\\\":23300812.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0732,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:35.3543979Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.975,\\\\\\\"memory\\\\\\\":23304432.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0617,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:45.3560901Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.387,\\\\\\\"memory\\\\\\\":23307736.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1176,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:55.3718002Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.442,\\\\\\\"memory\\\\\\\":23304548.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1254,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:22:05.3874211Z\\\\\\\",\\\\\\\"cpu\\\\\\\":7.766,\\\\\\\"memory\\\\\\\":23297692.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1201,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132}]}\\\\r\\\\nRequestStart: 2026-11-12T02:22:03.6687109Z; ResponseTime: 2026-11-12T02:22:09.6687149Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be187.documents.azure.com:11005/apps/457b0b96-1859-441b-9897-a8af2244ff42/services/0882f1b4-9e5e-44a9-a23d-5aff02eaa165/partitions/d4426837-d7ca-44b7-a111-280431846736/replicas/134337933848286181p, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5994.097, ActivityId: b3cae6cf-1e2a-46fe-90b3-6a504a79fd6f, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11005 | status: Connected | lkt: 11/12/2026 2:22:09 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6687109Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0052},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6687161Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0176},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6687337Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.3427},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6690764Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5995.1448},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:09.6642212Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1011},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:09.6643223Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-12T02:19:50.3213621Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-12T02:19:50.3213621Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-12T02:19:55.6660542Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-11T20:21:34.625Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T20:21:36.427Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:21:38.522Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:21:42.812Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:21:51.669Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:22:09.004Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:22:43.129Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:22:43.130Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T20:22:44.191Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0985344%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f
time=2026-11-11T20:30:46.072Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f/operationStatuses/08584071783219952338\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b161dda7-fd92-46b2-b2e8-4ab8c9119d2c\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b161dda7-fd92-46b2-b2e8-4ab8c9119d2c\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0985344/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j0985344\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: d1ede56b-38f5-4425-b382-a754f12b45fa, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: db332a6f-d6ca-4b3d-9da4-e65e2b4782a6, Request URI: /apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/326dacfa-a565-4b95-ab41-35500c782898/partitions/1a776f41-28f7-4e9f-8cb1-4d48f482ec6d/replicas/134384651191442271s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-11T20:29:52.3082641Z, RequestEndTime: 2026-11-11T20:29:57.3086766Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:06.3166707Z\\\\\\\",\\\\\\\"cpu\\\\\\\":7.488,\\\\\\\"memory\\\\\\\":23682836.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1221,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:16.3324546Z\\\\\\\",\\\\\\\"cpu\\\\\\\":4.524,\\\\\\\"memory\\\\\\\":23665912.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0709,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:26.3382658Z\\\\\\\",\\\\\\\"cpu\\\\\\\":4.219,\\\\\\\"memory\\\\\\\":23659192.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0824,\\\\\\\"availableThreads\\\\\\\":32762,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:36.3540698Z\\\\\\\",\\\\\\\"cpu\\\\\\\":7.313,\\\\\\\"memory\\\\\\\":23651792.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0979,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:46.3550310Z\\\\\\\",\\\\\\\"cpu\\\\\\\":16.367,\\\\\\\"memory\\\\\\\":23598600.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0665,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:56.3555696Z\\\\\\\",\\\\\\\"cpu\\\\\\\":5.060,\\\\\\\"memory\\\\\\\":23606476.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.096,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122}]}\\\\r\\\\nRequestStart: 2026-11-11T20:29:52.3082641Z; ResponseTime: 2026-11-11T20:29:57.3086766Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be149.documents.azure.com:11312/apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/326dacfa-a565-4b95-ab41-35500c782898/partitions/1a776f41-28f7-4e9f-8cb1-4d48f482ec6d/replicas/134384651191442271s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5002.075, ActivityId: db332a6f-d6ca-4b3d-9da4-e65e2b4782a6, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11312 | status: Connected | lkt: 11/11/2026 8:29:57 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3082641Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0037},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3082678Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.02},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3082878Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.2288},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3085166Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5002.673},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:57.3111896Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1463},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:57.3113359Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-11T20:27:38.9692278Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-11T20:27:38.9692278Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-11T20:27:39.0004715Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-10T11:16:22.539Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-10T11:16:24.181Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:26.335Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:30.679Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:39.545Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:55.947Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:17:31.257Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:17:31.257Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-10T11:17:31.890Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7760256%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10
time=2026-11-10T11:24:03.481Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10/operationStatuses/08584072974339535323\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e9565745-82ab-4caf-abba-d4474320716f\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e9565745-82ab-4caf-abba-d4474320716f\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7760256/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j7760256\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: e6bf8d0e-46dd-459a-9496-7274d225014a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 03242ad1-50e6-4647-a472-8bcee3bf6ab9, Request URI: /apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/285f097a-abb1-4c88-9d14-c42518f81a10/partitions/0f0d6a3c-7223-4cbb-a460-003329156d27/replicas/134383683272148038s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-10T11:23:08.2315094Z, RequestEndTime: 2026-11-10T11:23:13.2160337Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:20.7122803Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.346,\\\\\\\"memory\\\\\\\":23804772.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0467,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:30.7282997Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.579,\\\\\\\"memory\\\\\\\":23813272.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.096,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:40.7301913Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.109,\\\\\\\"memory\\\\\\\":23816912.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0604,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:50.7460385Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.814,\\\\\\\"memory\\\\\\\":23813536.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.065,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:23:00.7617917Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.755,\\\\\\\"memory\\\\\\\":23802572.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0864,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:23:10.7628025Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.348,\\\\\\\"memory\\\\\\\":23788572.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0913,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94}]}\\\\r\\\\nRequestStart: 2026-11-10T11:23:08.2315094Z; ResponseTime: 2026-11-10T11:23:13.2160337Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be149.documents.azure.com:11300/apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/285f097a-abb1-4c88-9d14-c42518f81a10/partitions/0f0d6a3c-7223-4cbb-a460-003329156d27/replicas/134383683272148038s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 4987.977, ActivityId: 03242ad1-50e6-4647-a472-8bcee3bf6ab9, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11300 | status: Connected | lkt: 11/10/2026 11:23:13 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2315094Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0028},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2315122Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0085},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2315207Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0911},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2316118Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 4988.5594},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:13.2201712Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1192},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:13.2202904Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-10T11:20:55.0207558Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-10T11:20:55.0207558Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-10T11:21:00.2239532Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster placement did not complete before the deadline; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message cluster placement did not complete before the deadline; provider Microsoft.RedHatOpenShift Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | e2e | 3 | 1.20%3 of 251 job runs affected | RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 5 · Nov 11: 16 · Nov 12: 31 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:124]: failed to create fully private HCP cluster "full-priv-kv" with private KV
Unexpected error:
<*fmt.wrapError | 0x2de5b2594880>:
failed waiting for cluster="full-priv-kv" in resourcegroup="full-priv-kv-4ccjmts62txv" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"name": "a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"status": "Failed",
"startTime": "2026-11-12T15:34:34.191986492Z",
"endTime": "2026-11-12T16:04:22.917130532Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
correlation ID: 11162cd7-dc7c-4515-9a6f-6e443f7b1a95
client request ID: 89f08d01-5d98-4385-a502-d2c0f1b98c7f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:124]: failed to create fully private HCP cluster "full-priv-kv" with private KV
Unexpected error:
<*fmt.wrapError | 0x2de5b2594880>:
failed waiting for cluster="full-priv-kv" in resourcegroup="full-priv-kv-4ccjmts62txv" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"name": "a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"status": "Failed",
"startTime": "2026-11-12T15:34:34.191986492Z",
"endTime": "2026-11-12T16:04:22.917130532Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
correlation ID: 11162cd7-dc7c-4515-9a6f-6e443f7b1a95
client request ID: 89f08d01-5d98-4385-a502-d2c0f1b98c7f
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-h6k5jr" with version 4.22.21 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3d89aea50b00>:
failed to create HCP cluster cluster-zstream-4-22-h6k5jr: failed waiting for cluster="cluster-zstream-4-22-h6k5jr" in resourcegroup="rg-zstream-upgrade-4-22-vdsknjqsrbfz" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"name": "7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"status": "Failed",
"startTime": "2026-11-12T15:33:13.01421483Z",
"endTime": "2026-11-12T16:07:07.952160345Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
correlation ID: 314f2449-0ae2-47ef-b9fd-89f78927292e
client request ID: 92baffe5-5311-4a79-a62c-25eb2f9fb38c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-h6k5jr" with version 4.22.21 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3d89aea50b00>:
failed to create HCP cluster cluster-zstream-4-22-h6k5jr: failed waiting for cluster="cluster-zstream-4-22-h6k5jr" in resourcegroup="rg-zstream-upgrade-4-22-vdsknjqsrbfz" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"name": "7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"status": "Failed",
"startTime": "2026-11-12T15:33:13.01421483Z",
"endTime": "2026-11-12T16:07:07.952160345Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
correlation ID: 314f2449-0ae2-47ef-b9fd-89f78927292e
client request ID: 92baffe5-5311-4a79-a62c-25eb2f9fb38c
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tfw224" with version 4.23.0-ec.1 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x377271de84a0>:
failed to create HCP cluster cluster-zstream-4-23-tfw224: failed waiting for cluster="cluster-zstream-4-23-tfw224" in resourcegroup="rg-zstream-upgrade-4-23-bn224qrf75xd" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec",
"name": "22c29099-f38e-42a3-9a4c-90084ad155ec",
"status": "Failed",
"startTime": "2026-11-12T15:32:40.053212969Z",
"endTime": "2026-11-12T16:01:50.625591398Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
correlation ID: 40d57c20-8bbf-48b4-8792-08c7b3124280
client request ID: 48105e19-c842-4e4b-bcc5-da59cee3b1bf
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tfw224" with version 4.23.0-ec.1 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x377271de84a0>:
failed to create HCP cluster cluster-zstream-4-23-tfw224: failed waiting for cluster="cluster-zstream-4-23-tfw224" in resourcegroup="rg-zstream-upgrade-4-23-bn224qrf75xd" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec",
"name": "22c29099-f38e-42a3-9a4c-90084ad155ec",
"status": "Failed",
"startTime": "2026-11-12T15:32:40.053212969Z",
"endTime": "2026-11-12T16:01:50.625591398Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
correlation ID: 40d57c20-8bbf-48b4-8792-08c7b3124280
client request ID: 48105e19-c842-4e4b-bcc5-da59cee3b1bf
occurredContributing tests (40)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to wait for first cluster "<cluster>" to complete creation (timeout '30.000000' minutes)full failure pattern: failed to wait for first cluster "<cluster>" to complete creation (timeout '30.000000' minutes) Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | e2e | 3 | 1.20%3 of 251 job runs affected | RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:132]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:132]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 65 · Nov 11: 54 · Nov 12: 21 · Nov 13: 0 · Nov 14: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapErrors | 0x1923f3bc0b70>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jn9zxd8bpzrt" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-jn9zxd8bpzrt\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1923f3d1cb00>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapErrors | 0x1923f3bc0b70>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jn9zxd8bpzrt" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-jn9zxd8bpzrt\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1923f3d1cb00>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapErrors | 0x3a9a18474690>:
failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-jkq8brgglgg7" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded
{
msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7",
},
<*fmt.wrapErrors | 0x3a9a18474660>{
msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>0x3a9a185ffe50 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapErrors | 0x3a9a18474690>:
failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-jkq8brgglgg7" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded
{
msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7",
},
<*fmt.wrapErrors | 0x3a9a18474660>{
msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>0x3a9a185ffe50 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapErrors | 0x31d885afe990>:
failed to create HCP cluster ingress-cert-le, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-xtr76b55krt4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: context deadline exceeded
{
msg: "failed to create HCP cluster ingress-cert-le, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: failed waiting for cluster=\"ingress-cert-le\" in resourcegroup=\"ingress-cert-le-xtr76b55krt4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x31d8861262f0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4",
},
<*fmt.wrapErrors | 0x31d885afe960>{
msg: "failed waiting for cluster=\"ingress-cert-le\" in resourcegroup=\"ingress-cert-le-xtr76b55krt4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x31d8861262f0>0x31d8861262f0 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapErrors | 0x31d885afe990>:
failed to create HCP cluster ingress-cert-le, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-xtr76b55krt4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: context deadline exceeded
{
msg: "failed to create HCP cluster ingress-cert-le, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: failed waiting for cluster=\"ingress-cert-le\" in resourcegroup=\"ingress-cert-le-xtr76b55krt4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x31d8861262f0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4",
},
<*fmt.wrapErrors | 0x31d885afe960>{
msg: "failed waiting for cluster=\"ingress-cert-le\" in resourcegroup=\"ingress-cert-le-xtr76b55krt4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901FromParam for cluster ingress-cert-le in resource group ingress-cert-le-xtr76b55krt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x31d8861262f0>0x31d8861262f0 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (68)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-11-12T23:02:19Z Ended: 2026-11-12T23:11:19Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2715136-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-11-12T23:02:19Z Ended: 2026-11-12T23:11:19Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2715136-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-11T02:18:36Z Ended: 2026-11-11T02:34:34Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4684288-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-09T10:44:19Z Ended: 2026-11-09T10:56:21Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5391488-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 7: 1 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 2 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2079e6d0e3e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
err: <*verifiers.pollObservationError | 0x2079e6d0e360>{
observation: <*errors.errorString | 0x2079e699e7c0>{
s: "dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
},
interruption: <*fmt.wrapError | 0x2079e6d0e340>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2079e6b8e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2079e6d0e300>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2079e80a8180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2079e6d0e3e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
err: <*verifiers.pollObservationError | 0x2079e6d0e360>{
observation: <*errors.errorString | 0x2079e699e7c0>{
s: "dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
},
interruption: <*fmt.wrapError | 0x2079e6d0e340>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2079e6b8e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2079e6d0e300>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2079e80a8180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x24dd696984c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
err: <*verifiers.pollObservationError | 0x24dd696984a0>{
observation: <*errors.errorString | 0x24dd688c93b0>{
s: "dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
},
interruption: <*fmt.wrapError | 0x24dd69698480>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x24dd6875c140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x24dd69698460>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x24dd6a0bc360>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x24dd696984c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
err: <*verifiers.pollObservationError | 0x24dd696984a0>{
observation: <*errors.errorString | 0x24dd688c93b0>{
s: "dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
},
interruption: <*fmt.wrapError | 0x24dd69698480>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x24dd6875c140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x24dd69698460>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x24dd6a0bc360>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2c89aca367e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
err: <*errors.errorString | 0x2c89ad98cef0>{
s: "dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2c89aca367e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
err: <*errors.errorString | 0x2c89ad98cef0>{
s: "dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
},
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2387254f8000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x238726e308c0>{
s: "dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2387254f8000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x238726e308c0>{
s: "dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1eea1ef05600>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
err: <*verifiers.pollObservationError | 0x1eea1ef055e0>{
observation: <*errors.errorString | 0x1eea20b3b080>{
s: "dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
},
interruption: <*fmt.wrapError | 0x1eea1ef055c0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1eea1ee206e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1eea1ef05560>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1eea2040eea0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1eea1ef05600>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
err: <*verifiers.pollObservationError | 0x1eea1ef055e0>{
observation: <*errors.errorString | 0x1eea20b3b080>{
s: "dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
},
interruption: <*fmt.wrapError | 0x1eea1ef055c0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1eea1ee206e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1eea1ef05560>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1eea2040eea0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x31589f922000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
err: <*errors.errorString | 0x31589ee10950>{
s: "dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x31589f922000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
err: <*errors.errorString | 0x31589ee10950>{
s: "dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
},
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [svc] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-12T16:02:47Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-jpbc4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-12T16:02:48Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-jpbc4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-11T19:13:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-hbqnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-11T19:13:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-hbqnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-10T22:16:06Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9699200-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9699200/providers/microsoft.monitor/accounts/services-j9699200", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-vmhll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-10T22:16:06Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9699200-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9699200/providers/microsoft.monitor/accounts/services-j9699200", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-vmhll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | alert | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 6 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-12T15:52:46Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-j4rtb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f51ff0e8-0932-44b5-86a1-12ea492c373f" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-j4rtb (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-11-12T15:52:46Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-j4rtb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f51ff0e8-0932-44b5-86a1-12ea492c373f" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-j4rtb (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-11-12T16:01:46Z Ended: 2026-11-12T16:57:57Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Resolved Started: 2026-11-12T16:01:46Z Ended: 2026-11-12T16:57:48Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 5: State: Fired (not resolved) Started: 2026-11-12T17:13:47Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 6: State: Fired (not resolved) Started: 2026-11-12T17:13:47Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). alert fired 6 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-11T19:17:09Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-88b69", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="468e8019-4d61-44d8-a033-41db4b1a7b1b" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-88b69 (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-11-11T19:17:09Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-88b69", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="468e8019-4d61-44d8-a033-41db4b1a7b1b" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-88b69 (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-11-11T19:29:14Z Ended: 2026-11-11T19:35:10Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Resolved Started: 2026-11-11T19:29:14Z Ended: 2026-11-11T19:35:10Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 5: State: Fired (not resolved) Started: 2026-11-11T20:34:08Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 6: State: Fired (not resolved) Started: 2026-11-11T20:34:08Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-10T22:08:06Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9699200-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9699200/providers/microsoft.monitor/accounts/services-j9699200", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-5dc7dbc48f-8ltng", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4cedcebe-352d-4bc7-9f9a-258e12387129" Description: Pod aro-hcp/aro-hcp-backend-5dc7dbc48f-8ltng (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-11-10T22:09:05Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9699200-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9699200/providers/microsoft.monitor/accounts/services-j9699200", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-5dc7dbc48f-8ltng", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4cedcebe-352d-4bc7-9f9a-258e12387129" Description: Pod aro-hcp/aro-hcp-backend-5dc7dbc48f-8ltng (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-11-10T22:14:06Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9699200-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9699200/providers/microsoft.monitor/accounts/services-j9699200", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-5dc7dbc48f-nrjrt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="87176d61-26f9-4685-8c31-e46cc3231c3a" Description: Pod aro-hcp/aro-hcp-backend-5dc7dbc48f-nrjrt (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-11-10T22:14:06Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9699200-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9699200/providers/microsoft.monitor/accounts/services-j9699200", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-5dc7dbc48f-nrjrt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="87176d61-26f9-4685-8c31-e46cc3231c3a" Description: Pod aro-hcp/aro-hcp-backend-5dc7dbc48f-nrjrt (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
scheduling requested CPU must be positivefull failure pattern: scheduling requested CPU must be positive Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.20%3 of 251 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
<bool>: true
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
<bool>: true
to be falsefail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 1 scheduling requested CPU must be positive
Expected
<bool>: true
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 1 scheduling requested CPU must be positive
Expected
<bool>: true
to be falseContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | alert | 3 | 1.20%3 of 251 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 2 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 15 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-wdpfr", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-wdpfr has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000001-hj882", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000001-hj882 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-5cffdb78dc-tz654", region="canadacentral", severity="warning" Description: Pod kube-system/hubble-relay-5cffdb78dc-tz654 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000002-5w7z2", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000002-5w7z2 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-cxdpf", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-cxdpf has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-vxj68", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-vxj68 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000000-84std", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000000-84std has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-t6cxk", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-t6cxk has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-sqzqm", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-sqzqm has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-gb9rv", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-gb9rv has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-78789f646f-2wdlw", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-78789f646f-2wdlw has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-78789f646f-z5ntg", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-78789f646f-z5ntg has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-wr8wf", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-wr8wf has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-zncb7", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-zncb7 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-11-13T20:44:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-9vgxw", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-9vgxw has been in a non-ready state for longer than 5 minutes. alert fired 3 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-13T11:18:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2107008-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-585c6ccf64-r8d9p", region="centralus", severity="warning" Description: Pod clusters-service/clusters-service-585c6ccf64-r8d9p has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-13T11:18:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2107008-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-585c6ccf64-ghmv8", region="centralus", severity="warning" Description: Pod clusters-service/clusters-service-585c6ccf64-ghmv8 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-11-13T11:18:22Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2107008-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-585c6ccf64-nl8b5", region="centralus", severity="warning" Description: Pod clusters-service/clusters-service-585c6ccf64-nl8b5 has been in a non-ready state for longer than 5 minutes. alert fired 97 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-hbnf5", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-hbnf5 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-5cddk", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-5cddk has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-hwfwj", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-hwfwj has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-54k4s", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-54k4s has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-6s579", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-6s579 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-dx4kd", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-dx4kd has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-kb485", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-kb485 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-z9c28", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-z9c28 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-mhvbn", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-mhvbn has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-jpn5p", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-jpn5p has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-gx2hr", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-gx2hr has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-zqdxm", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-zqdxm has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-777ds", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-777ds has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-9kbjv", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-9kbjv has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-j4ntc", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-j4ntc has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-prc22", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-prc22 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-hfsrq", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-hfsrq has been in a non-ready state for longer than 5 minutes. Firing 18: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-5xztv", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-5xztv has been in a non-ready state for longer than 5 minutes. Firing 19: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-vnnwm", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-vnnwm has been in a non-ready state for longer than 5 minutes. Firing 20: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-njlqc", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-njlqc has been in a non-ready state for longer than 5 minutes. Firing 21: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-c5ks7", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-c5ks7 has been in a non-ready state for longer than 5 minutes. Firing 22: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-f5m7h", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-f5m7h has been in a non-ready state for longer than 5 minutes. Firing 23: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-xmv2n", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-xmv2n has been in a non-ready state for longer than 5 minutes. Firing 24: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-c5zdw", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-c5zdw has been in a non-ready state for longer than 5 minutes. Firing 25: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-6jsh7", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-6jsh7 has been in a non-ready state for longer than 5 minutes. Firing 26: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-fcghz", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-fcghz has been in a non-ready state for longer than 5 minutes. Firing 27: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-s6rz5", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-s6rz5 has been in a non-ready state for longer than 5 minutes. Firing 28: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-ghh92", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-ghh92 has been in a non-ready state for longer than 5 minutes. Firing 29: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-shh7d", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-shh7d has been in a non-ready state for longer than 5 minutes. Firing 30: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-t9st7", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-t9st7 has been in a non-ready state for longer than 5 minutes. Firing 31: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-9krtb", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-9krtb has been in a non-ready state for longer than 5 minutes. Firing 32: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-qd655", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-qd655 has been in a non-ready state for longer than 5 minutes. Firing 33: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-jcpbr", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-jcpbr has been in a non-ready state for longer than 5 minutes. Firing 34: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-k2h2g", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-k2h2g has been in a non-ready state for longer than 5 minutes. Firing 35: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-9q4b9", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-9q4b9 has been in a non-ready state for longer than 5 minutes. Firing 36: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-982vv", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-982vv has been in a non-ready state for longer than 5 minutes. Firing 37: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-h5qpg", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-h5qpg has been in a non-ready state for longer than 5 minutes. Firing 38: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-l98vb", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-l98vb has been in a non-ready state for longer than 5 minutes. Firing 39: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-8lb2c", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-8lb2c has been in a non-ready state for longer than 5 minutes. Firing 40: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-m8slg", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-m8slg has been in a non-ready state for longer than 5 minutes. Firing 41: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-v8w4w", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-v8w4w has been in a non-ready state for longer than 5 minutes. Firing 42: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-5n46c", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-5n46c has been in a non-ready state for longer than 5 minutes. Firing 43: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-xfvhp", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-xfvhp has been in a non-ready state for longer than 5 minutes. Firing 44: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-lw2gf", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-lw2gf has been in a non-ready state for longer than 5 minutes. Firing 45: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-xj985", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-xj985 has been in a non-ready state for longer than 5 minutes. Firing 46: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-krs9z", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-krs9z has been in a non-ready state for longer than 5 minutes. Firing 47: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-2fgs6", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-2fgs6 has been in a non-ready state for longer than 5 minutes. Firing 48: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-jftlz", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-jftlz has been in a non-ready state for longer than 5 minutes. Firing 49: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-x8k9x", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-x8k9x has been in a non-ready state for longer than 5 minutes. Firing 50: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-zbvx6", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-zbvx6 has been in a non-ready state for longer than 5 minutes. Firing 51: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-pvsfz", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-pvsfz has been in a non-ready state for longer than 5 minutes. Firing 52: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-lk9sm", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-lk9sm has been in a non-ready state for longer than 5 minutes. Firing 53: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-wvx2x", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-wvx2x has been in a non-ready state for longer than 5 minutes. Firing 54: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-d9jtt", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-d9jtt has been in a non-ready state for longer than 5 minutes. Firing 55: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-k9v6d", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-k9v6d has been in a non-ready state for longer than 5 minutes. Firing 56: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-2hf2h", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-2hf2h has been in a non-ready state for longer than 5 minutes. Firing 57: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-kn25w", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-kn25w has been in a non-ready state for longer than 5 minutes. Firing 58: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-rq5c4", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-rq5c4 has been in a non-ready state for longer than 5 minutes. Firing 59: State: Fired (not resolved) Started: 2026-11-12T22:58:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-d9qp9", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-d9qp9 has been in a non-ready state for longer than 5 minutes. Firing 60: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-m9llx", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-m9llx has been in a non-ready state for longer than 5 minutes. Firing 61: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-lmqgz", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-lmqgz has been in a non-ready state for longer than 5 minutes. Firing 62: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-rj8jl", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-rj8jl has been in a non-ready state for longer than 5 minutes. Firing 63: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-6v7pv", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-6v7pv has been in a non-ready state for longer than 5 minutes. Firing 64: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-xqmw4", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-xqmw4 has been in a non-ready state for longer than 5 minutes. Firing 65: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-q882x", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-q882x has been in a non-ready state for longer than 5 minutes. Firing 66: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-4v55b", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-4v55b has been in a non-ready state for longer than 5 minutes. Firing 67: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-dzmr8", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-dzmr8 has been in a non-ready state for longer than 5 minutes. Firing 68: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-lrltc", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-lrltc has been in a non-ready state for longer than 5 minutes. Firing 69: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-gc85r", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-gc85r has been in a non-ready state for longer than 5 minutes. Firing 70: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-htf2q", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-htf2q has been in a non-ready state for longer than 5 minutes. Firing 71: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-dt4k5", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-dt4k5 has been in a non-ready state for longer than 5 minutes. Firing 72: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6d55bbbd48-27nc9", region="centralus", severity="warning" Description: Pod clusters-service/clusters-service-6d55bbbd48-27nc9 has been in a non-ready state for longer than 5 minutes. Firing 73: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-szjg9", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-szjg9 has been in a non-ready state for longer than 5 minutes. Firing 74: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-fhhqf", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-fhhqf has been in a non-ready state for longer than 5 minutes. Firing 75: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-n9cx8", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-n9cx8 has been in a non-ready state for longer than 5 minutes. Firing 76: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-9xcwd", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-9xcwd has been in a non-ready state for longer than 5 minutes. Firing 77: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-8vs74", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-8vs74 has been in a non-ready state for longer than 5 minutes. Firing 78: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-frvx6", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-frvx6 has been in a non-ready state for longer than 5 minutes. Firing 79: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-czq4k", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-czq4k has been in a non-ready state for longer than 5 minutes. Firing 80: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-9tv5f", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-9tv5f has been in a non-ready state for longer than 5 minutes. Firing 81: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-ph5b9", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-ph5b9 has been in a non-ready state for longer than 5 minutes. Firing 82: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-5fpm9", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-5fpm9 has been in a non-ready state for longer than 5 minutes. Firing 83: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-4nbvh", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-4nbvh has been in a non-ready state for longer than 5 minutes. Firing 84: State: Fired (not resolved) Started: 2026-11-12T22:58:27Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-shg5p", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-shg5p has been in a non-ready state for longer than 5 minutes. Firing 85: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-8gcd4", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-8gcd4 has been in a non-ready state for longer than 5 minutes. Firing 86: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-zd5cm", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-zd5cm has been in a non-ready state for longer than 5 minutes. Firing 87: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-5vznh", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-5vznh has been in a non-ready state for longer than 5 minutes. Firing 88: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-rvgd9", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-rvgd9 has been in a non-ready state for longer than 5 minutes. Firing 89: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-brzbr", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-brzbr has been in a non-ready state for longer than 5 minutes. Firing 90: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-7t2t4", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-7t2t4 has been in a non-ready state for longer than 5 minutes. Firing 91: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-hb9hm", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-hb9hm has been in a non-ready state for longer than 5 minutes. Firing 92: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-zk4lc", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-zk4lc has been in a non-ready state for longer than 5 minutes. Firing 93: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-5lq9q", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-5lq9q has been in a non-ready state for longer than 5 minutes. Firing 94: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-gzrng", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-gzrng has been in a non-ready state for longer than 5 minutes. Firing 95: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-tvknw", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-tvknw has been in a non-ready state for longer than 5 minutes. Firing 96: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-h28g7", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-h28g7 has been in a non-ready state for longer than 5 minutes. Firing 97: State: Fired (not resolved) Started: 2026-11-12T22:59:26Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5922560-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-6fd7886dff-pptfs", region="centralus", severity="warning" Description: Pod sessiongate/sessiongate-6fd7886dff-pptfs has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 80.00% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster then delete it by deleting the customer resource group | |
| 84.44% | 45 | integration/parallel | Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully | |
| 84.44% | 45 | integration/parallel | Customer should be able to update node pool labels and taints |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 2 | 9.09%2 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 7 · Nov 13: 0 · Nov 14: 0 | STG | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:99]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x17b1600e4de0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-psp6ghkd2r7p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"name": "cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"status": "Failed",
"startTime": "2026-11-12T19:48:15.197021893Z",
"endTime": "2026-11-12T20:17:16.216474731Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 25m13s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
correlation ID: 588080de-4376-4b27-928a-03d587d57b2d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:99]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x17b1600e4de0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-psp6ghkd2r7p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"name": "cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"status": "Failed",
"startTime": "2026-11-12T19:48:15.197021893Z",
"endTime": "2026-11-12T20:17:16.216474731Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 25m13s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
correlation ID: 588080de-4376-4b27-928a-03d587d57b2d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-rqw6dbbkwpq8/cluster-candidate-5-1-25sx5t should provision
Unexpected error:
<*fmt.wrapError | 0x2f3a166b8800>:
failed to create HCP cluster cluster-candidate-5-1-25sx5t: failed waiting for cluster="cluster-candidate-5-1-25sx5t" in resourcegroup="rg-candidate-5-1-rqw6dbbkwpq8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489",
"name": "0ad08df1-5222-482b-8e92-25a4d0649489",
"status": "Failed",
"startTime": "2026-11-12T19:48:14.366989654Z",
"endTime": "2026-11-12T20:17:15.941327806Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 23m10s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
correlation ID: f77ffdcc-6db4-4939-bd7d-f626c9d73793
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-rqw6dbbkwpq8/cluster-candidate-5-1-25sx5t should provision
Unexpected error:
<*fmt.wrapError | 0x2f3a166b8800>:
failed to create HCP cluster cluster-candidate-5-1-25sx5t: failed waiting for cluster="cluster-candidate-5-1-25sx5t" in resourcegroup="rg-candidate-5-1-rqw6dbbkwpq8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489",
"name": "0ad08df1-5222-482b-8e92-25a4d0649489",
"status": "Failed",
"startTime": "2026-11-12T19:48:14.366989654Z",
"endTime": "2026-11-12T20:17:15.941327806Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 23m10s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
correlation ID: f77ffdcc-6db4-4939-bd7d-f626c9d73793
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x28e2f0a88a80>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nfkbd8xd9kxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5",
"name": "324ccd68-ef42-41e9-a980-fd643a150df5",
"status": "Failed",
"startTime": "2026-11-12T17:11:50.950828556Z",
"endTime": "2026-11-12T17:40:56.295707627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 20m27s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
correlation ID: a04ff94c-430d-45c3-978d-87b26669fcf4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x28e2f0a88a80>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nfkbd8xd9kxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5",
"name": "324ccd68-ef42-41e9-a980-fd643a150df5",
"status": "Failed",
"startTime": "2026-11-12T17:11:50.950828556Z",
"endTime": "2026-11-12T17:40:56.295707627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 20m27s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
correlation ID: a04ff94c-430d-45c3-978d-87b26669fcf4
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 2 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:100]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
<*fmt.wrapError | 0x1b4614d815e0>:
failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pk4g7jxvwskz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"name": "5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"status": "Failed",
"startTime": "2026-11-12T17:11:52.373940715Z",
"endTime": "2026-11-12T17:40:56.797994555Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 22m16s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
correlation ID: ed91409f-3f3e-40f0-af6f-e1efc0a6cad7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:100]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
<*fmt.wrapError | 0x1b4614d815e0>:
failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pk4g7jxvwskz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"name": "5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"status": "Failed",
"startTime": "2026-11-12T17:11:52.373940715Z",
"endTime": "2026-11-12T17:40:56.797994555Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 22m16s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
correlation ID: ed91409f-3f3e-40f0-af6f-e1efc0a6cad7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-9qmh46" with version 5.0.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3c857efdd5e0>:
failed to create HCP cluster cluster-zstream-5-0-9qmh46: failed waiting for cluster="cluster-zstream-5-0-9qmh46" in resourcegroup="rg-zstream-upgrade-5-0-2dkdlwm4dztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e",
"name": "74ae707c-8d62-4995-b262-0153bb8bf32e",
"status": "Failed",
"startTime": "2026-11-12T17:11:53.237843141Z",
"endTime": "2026-11-12T17:40:56.804735597Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0 is Partial (want Completed), started 22m14s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
correlation ID: b786b9a5-3d9a-4d46-bae8-4f64d6429da9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-9qmh46" with version 5.0.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3c857efdd5e0>:
failed to create HCP cluster cluster-zstream-5-0-9qmh46: failed waiting for cluster="cluster-zstream-5-0-9qmh46" in resourcegroup="rg-zstream-upgrade-5-0-2dkdlwm4dztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e",
"name": "74ae707c-8d62-4995-b262-0153bb8bf32e",
"status": "Failed",
"startTime": "2026-11-12T17:11:53.237843141Z",
"endTime": "2026-11-12T17:40:56.804735597Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0 is Partial (want Completed), started 22m14s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
correlation ID: b786b9a5-3d9a-4d46-bae8-4f64d6429da9
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceededfull failure pattern: DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x3311a2f0c120>:
resource group "rg-candidate-4-23-kk2s78nmqwx7": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7", caused by: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x3311a20263e0>{
msg: "resource group \"rg-candidate-4-23-kk2s78nmqwx7\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapError | 0x3311a20263c0>{
msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapErrors | 0x3311a23a85a0>{
msg: "failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*fmt.wrapErrors | 0x3311a23a8570>{
msg: "failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3311a2366650>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7",
},
<context.deadlineExceededError>{},
],
},
<*fmt.wrapErrors | 0x3311a23a8570>0x3311a23a8570 (cyclic reference),
],
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x3311a2f0c120>:
resource group "rg-candidate-4-23-kk2s78nmqwx7": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7", caused by: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x3311a20263e0>{
msg: "resource group \"rg-candidate-4-23-kk2s78nmqwx7\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapError | 0x3311a20263c0>{
msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapErrors | 0x3311a23a85a0>{
msg: "failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*fmt.wrapErrors | 0x3311a23a8570>{
msg: "failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3311a2366650>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7",
},
<context.deadlineExceededError>{},
],
},
<*fmt.wrapErrors | 0x3311a23a8570>0x3311a23a8570 (cyclic reference),
],
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-m7c8jjt7kz5k/cluster-candidate-5-1-zwr77b should provision
Unexpected error:
<*fmt.wrapError | 0x317411260>:
failed to create HCP cluster cluster-candidate-5-1-zwr77b: failed waiting for cluster="cluster-candidate-5-1-zwr77b" in resourcegroup="rg-candidate-5-1-m7c8jjt7kz5k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93",
"name": "d222ac87-ad5a-4334-8130-3c30fedb6e93",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.805632783Z",
"endTime": "2026-11-12T17:40:46.399524896Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 20m19s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
correlation ID: 6a614cae-828b-4e97-ac1e-2aac8b7ca153
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-m7c8jjt7kz5k/cluster-candidate-5-1-zwr77b should provision
Unexpected error:
<*fmt.wrapError | 0x317411260>:
failed to create HCP cluster cluster-candidate-5-1-zwr77b: failed waiting for cluster="cluster-candidate-5-1-zwr77b" in resourcegroup="rg-candidate-5-1-m7c8jjt7kz5k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93",
"name": "d222ac87-ad5a-4334-8130-3c30fedb6e93",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.805632783Z",
"endTime": "2026-11-12T17:40:46.399524896Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 20m19s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
correlation ID: 6a614cae-828b-4e97-ac1e-2aac8b7ca153
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x26c329a4a8e0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rk97m722fgxr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"name": "6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"status": "Failed",
"startTime": "2026-11-12T17:14:14.481992445Z",
"endTime": "2026-11-12T17:43:16.629667554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, packageserver, cluster-network-operator, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, ingress-operator, redhat-operators-catalog, openshift-controller-manager, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog, catalog-operator, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
correlation ID: c03cdfce-2433-48c4-b0bf-dd2331b70f7a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x26c329a4a8e0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rk97m722fgxr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"name": "6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"status": "Failed",
"startTime": "2026-11-12T17:14:14.481992445Z",
"endTime": "2026-11-12T17:43:16.629667554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, packageserver, cluster-network-operator, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, ingress-operator, redhat-operators-catalog, openshift-controller-manager, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog, catalog-operator, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
correlation ID: c03cdfce-2433-48c4-b0bf-dd2331b70f7a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-rnvnjl with version 4.21.42
Unexpected error:
<*fmt.wrapError | 0x33618be5f5c0>:
failed to create HCP cluster np-version-upgrade-cluster-rnvnjl: failed waiting for cluster="np-version-upgrade-cluster-rnvnjl" in resourcegroup="rg-np-version-upgrade-rnvnjl-khbkdtxdkfdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a",
"name": "3f68db84-4a8d-4bf7-9849-3e7db026805a",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.409592604Z",
"endTime": "2026-11-12T17:40:46.473882071Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
correlation ID: 2c6cdab0-a3af-4588-8f80-5ebc352de1b7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-rnvnjl with version 4.21.42
Unexpected error:
<*fmt.wrapError | 0x33618be5f5c0>:
failed to create HCP cluster np-version-upgrade-cluster-rnvnjl: failed waiting for cluster="np-version-upgrade-cluster-rnvnjl" in resourcegroup="rg-np-version-upgrade-rnvnjl-khbkdtxdkfdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a",
"name": "3f68db84-4a8d-4bf7-9849-3e7db026805a",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.409592604Z",
"endTime": "2026-11-12T17:40:46.473882071Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
correlation ID: 2c6cdab0-a3af-4588-8f80-5ebc352de1b7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining res...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x142a3dd04180>:
resource group "rg-candidate-4-23-cxmw6dwj8lvz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-l5472k" in resourcegroup="rg-candidate-4-23-cxmw6dwj8lvz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"name": "9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"status": "Failed",
"startTime": "2026-11-09T17:35:42.053088057Z",
"endTime": "2026-11-09T17:59:50.562259882Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
correlation ID: 97bda22f-2f83-4bf1-94bb-3865756e10f7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x142a3dd04180>:
resource group "rg-candidate-4-23-cxmw6dwj8lvz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-l5472k" in resourcegroup="rg-candidate-4-23-cxmw6dwj8lvz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"name": "9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"status": "Failed",
"startTime": "2026-11-09T17:35:42.053088057Z",
"endTime": "2026-11-09T17:59:50.562259882Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
correlation ID: 97bda22f-2f83-4bf1-94bb-3865756e10f7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-lftmdz-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-lftmdz-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-lftmdz-<id>": nodes "cluster-np-lftmdz-<id>" is forbidden Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-lftmdz"
Unexpected error:
<*errors.joinError | 0x2c80b1a3b560>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-lftmdz-ghqg5-5jcqq": nodes "cluster-np-lftmdz-ghqg5-5jcqq" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2c80b1940940>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940920>{
msg: "failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940900>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2c80b186d7c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-lftmdz-ghqg5-5jcqq",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-lftmdz"
Unexpected error:
<*errors.joinError | 0x2c80b1a3b560>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-lftmdz-ghqg5-5jcqq": nodes "cluster-np-lftmdz-ghqg5-5jcqq" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2c80b1940940>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940920>{
msg: "failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940900>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2c80b186d7c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-lftmdz-ghqg5-5jcqq",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp...full failure pattern: VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-97dm56"
Unexpected error:
<*fmt.wrapError | 0x1dbf7ed98e40>:
VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-fn9vpbv"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-fn9vpbv\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x1dbf809b21d0>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-97dm56"
Unexpected error:
<*fmt.wrapError | 0x1dbf7ed98e40>:
VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-fn9vpbv"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-fn9vpbv\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x1dbf809b21d0>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net",
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in name...full failure pattern: polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors: Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:346]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpull9vvdflkz2tvn.azurecr.io
Unexpected error:
<*fmt.wrapError | 0x1d5b57c7d1c0>:
image pull not confirmed; cause undetermined (insufficient evidence)
polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
- pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d
{
msg: "image pull not confirmed; cause undetermined (insufficient evidence)\npolling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*fmt.wrapError | 0x1d5b57c7d180>{
msg: "VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*verifiers.pollObservationError | 0x1d5b57c7d040>{
observation: <*errors.errorString | 0x1d5b59f3bdd0>{
s: "expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
},
interruption: <*fmt.wrapError | 0x1d5b57c7d020>{
msg: "failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:346]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpull9vvdflkz2tvn.azurecr.io
Unexpected error:
<*fmt.wrapError | 0x1d5b57c7d1c0>:
image pull not confirmed; cause undetermined (insufficient evidence)
polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
- pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d
{
msg: "image pull not confirmed; cause undetermined (insufficient evidence)\npolling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*fmt.wrapError | 0x1d5b57c7d180>{
msg: "VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*verifiers.pollObservationError | 0x1d5b57c7d040>{
observation: <*errors.errorString | 0x1d5b59f3bdd0>{
s: "expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
},
interruption: <*fmt.wrapError | 0x1d5b57c7d020>{
msg: "failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:153]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x19f475ced140>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c96mksh6wtzx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-c96mksh6wtzx\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x19f474e59970>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:153]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x19f475ced140>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c96mksh6wtzx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-c96mksh6wtzx\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x19f474e59970>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 92.86% | 14 | stage/parallel | Customer should be able to get a specific HCP Operator Identity Role Set by version | |
| 92.86% | 14 | stage/parallel | Customer should be able to list available HCP Operator Identity Role Sets and validate response content | |
| 92.86% | 14 | stage/parallel | Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 5.00%1 of 20 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | PROD | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:149]: failed to verify cluster "cp-ystream-upgrade-5-0-jgz7z5" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x3e96a1c86db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e96a1d21c80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e96a1d21c60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e96a28e4820>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:149]: failed to verify cluster "cp-ystream-upgrade-5-0-jgz7z5" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x3e96a1c86db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e96a1d21c80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e96a1d21c60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e96a28e4820>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 5.00%1 of 20 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x270039317a40>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-q2hwcpbx7mqn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220",
"name": "2f19f972-635b-47fe-bd8e-1577e0895220",
"status": "Failed",
"startTime": "2026-11-08T23:05:43.229286441Z",
"endTime": "2026-11-08T23:34:48.983552333Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
correlation ID: 2b542b89-9e3f-45d8-a8b2-acddb890ebb9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x270039317a40>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-q2hwcpbx7mqn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220",
"name": "2f19f972-635b-47fe-bd8e-1577e0895220",
"status": "Failed",
"startTime": "2026-11-08T23:05:43.229286441Z",
"endTime": "2026-11-08T23:34:48.983552333Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
correlation ID: 2b542b89-9e3f-45d8-a8b2-acddb890ebb9
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 5.00%1 of 20 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | INT | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x5b2334e0700>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wm9dc9s5xztk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153",
"name": "12b091af-f05e-4977-a8fb-00833f004153",
"status": "Failed",
"startTime": "2026-11-08T23:07:24.292926055Z",
"endTime": "2026-11-08T23:36:29.334124474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 11m56s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
correlation ID: ca2f27d7-78cd-47db-9589-31ad832f7406
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x5b2334e0700>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wm9dc9s5xztk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153",
"name": "12b091af-f05e-4977-a8fb-00833f004153",
"status": "Failed",
"startTime": "2026-11-08T23:07:24.292926055Z",
"endTime": "2026-11-08T23:36:29.334124474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 11m56s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
correlation ID: ca2f27d7-78cd-47db-9589-31ad832f7406
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "c...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xdkrt6-zjdgs-44h72": nodes "cluster-np-xdkrt6-zjdgs-44h72" is Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 5.00%1 of 20 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xdkrt6"
Unexpected error:
<*errors.joinError | 0x22f6aa118348>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xdkrt6-zjdgs-44h72": nodes "cluster-np-xdkrt6-zjdgs-44h72" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x22f6a8c83260>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83240>{
msg: "failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83220>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x22f6a8ca6fa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xdkrt6-zjdgs-44h72",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xdkrt6"
Unexpected error:
<*errors.joinError | 0x22f6aa118348>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xdkrt6-zjdgs-44h72": nodes "cluster-np-xdkrt6-zjdgs-44h72" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x22f6a8c83260>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83240>{
msg: "failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83220>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x22f6a8ca6fa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xdkrt6-zjdgs-44h72",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 46.15% | 13 | prod/parallel | Customer should be able to get a specific HCP Operator Identity Role Set by version | |
| 46.15% | 13 | prod/parallel | Customer should be able to list available HCP Operator Identity Role Sets and validate response content | |
| 92.31% | 13 | prod/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 92.31% | 13 | prod/parallel | Customer should create an HCP cluster and validate TLS certificates |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 5 | 7.69%5 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsNov 7: 10 · Nov 8: 8 · Nov 9: 2 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e16e18c95f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: e5e9735e-d3e5-4012-86ee-338d0c182df4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e16e18c95f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: e5e9735e-d3e5-4012-86ee-338d0c182df4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2aad74a796b0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: 38cd85f4-6402-47f3-975b-00cf55650f28
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2aad74a796b0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: 38cd85f4-6402-47f3-975b-00cf55650f28
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x364c7b3b3050>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 7f875e35-2e2b-44ef-9cc5-b907e9231d1e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x364c7b3b3050>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 7f875e35-2e2b-44ef-9cc5-b907e9231d1e
client request ID: unavailable
occurredContributing tests (6)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 3 | 4.62%3 of 65 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 1 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x5ec36ffd968>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5ec37183140>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5ec37183120>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5ec36a47680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5ec3701ed60>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5ec3701ed40>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5ec36c2fe00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x5ec36ffd968>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5ec37183140>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5ec37183120>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5ec36a47680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5ec3701ed60>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5ec3701ed40>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5ec36c2fe00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-9m6tns" is viable
Unexpected error:
<*errors.joinError | 0x2dc6f9ea210>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2dc6f0a5a80>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2dc6f0a5a60>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2dc6f286960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2dc6f2be280>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2dc6f2be260>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2dc6f3b35e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-9m6tns" is viable
Unexpected error:
<*errors.joinError | 0x2dc6f9ea210>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2dc6f0a5a80>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2dc6f0a5a60>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2dc6f286960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2dc6f2be280>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2dc6f2be260>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2dc6f3b35e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
<*errors.joinError | 0x21c2e72b6dc8>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21c2e7d1a4e0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21c2e7d1a4c0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21c2e7312280>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21c2e8078100>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21c2e80780e0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21c2e7732f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
<*errors.joinError | 0x21c2e72b6dc8>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21c2e7d1a4e0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21c2e7d1a4c0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21c2e7312280>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21c2e8078100>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21c2e80780e0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21c2e7732f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 4.62%3 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 3 · Nov 12: 0 · Nov 13: 6 · Nov 14: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7"
Unexpected error:
<*fmt.wrapError | 0x3eb94d03ed60>:
failed to create NodePool np-k5wcs7: failed waiting for nodepool="np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7" in resourcegroup="rg-candidate-4-23-fvhwkprkl5hk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926",
"name": "edad2cce-0c27-4141-af66-5421d0367926",
"status": "Failed",
"startTime": "2026-11-13T06:43:01.877199954Z",
"endTime": "2026-11-13T07:02:05.836906321Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: Creating: virtualmachine creating or updating\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
correlation ID: b6db496e-1ecf-4f45-98bf-19f14c9919e4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7"
Unexpected error:
<*fmt.wrapError | 0x3eb94d03ed60>:
failed to create NodePool np-k5wcs7: failed waiting for nodepool="np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7" in resourcegroup="rg-candidate-4-23-fvhwkprkl5hk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926",
"name": "edad2cce-0c27-4141-af66-5421d0367926",
"status": "Failed",
"startTime": "2026-11-13T06:43:01.877199954Z",
"endTime": "2026-11-13T07:02:05.836906321Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: Creating: virtualmachine creating or updating\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
correlation ID: b6db496e-1ecf-4f45-98bf-19f14c9919e4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x2c3a6de09320>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-pvc2kk" in resourcegroup="rg-np-noedge-pvc2kk-f5hxq2mnnx2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8",
"name": "f41964ee-de97-4466-b5e8-feb618fc82a8",
"status": "Failed",
"startTime": "2026-11-13T07:19:26.464221032Z",
"endTime": "2026-11-13T07:38:31.657256324Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-4wlgb-528bt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-4wlgb-528bt: Creating: virtualmachine creating or updating\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
correlation ID: e2a37064-75a7-4e62-90a9-71bfa7da63c5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x2c3a6de09320>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-pvc2kk" in resourcegroup="rg-np-noedge-pvc2kk-f5hxq2mnnx2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8",
"name": "f41964ee-de97-4466-b5e8-feb618fc82a8",
"status": "Failed",
"startTime": "2026-11-13T07:19:26.464221032Z",
"endTime": "2026-11-13T07:38:31.657256324Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-4wlgb-528bt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-4wlgb-528bt: Creating: virtualmachine creating or updating\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
correlation ID: e2a37064-75a7-4e62-90a9-71bfa7da63c5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz"
Unexpected error:
<*fmt.wrapError | 0x17145c623cc0>:
failed to create NodePool np-pwhdqz: failed waiting for nodepool="np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz" in resourcegroup="rg-candidate-5-1-z27fm55sbdhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd",
"name": "8994658d-0314-4930-b123-77a9aa8d6ddd",
"status": "Failed",
"startTime": "2026-11-13T06:47:16.516948762Z",
"endTime": "2026-11-13T07:06:16.339079825Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-pwhdqz-nrnj4-nllg8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-pwhdqz-nrnj4-nllg8: Creating: virtualmachine creating or updating\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
correlation ID: 67c21db5-2743-4fb5-a2e0-d37a965dd5f1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz"
Unexpected error:
<*fmt.wrapError | 0x17145c623cc0>:
failed to create NodePool np-pwhdqz: failed waiting for nodepool="np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz" in resourcegroup="rg-candidate-5-1-z27fm55sbdhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd",
"name": "8994658d-0314-4930-b123-77a9aa8d6ddd",
"status": "Failed",
"startTime": "2026-11-13T06:47:16.516948762Z",
"endTime": "2026-11-13T07:06:16.339079825Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-pwhdqz-nrnj4-nllg8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-pwhdqz-nrnj4-nllg8: Creating: virtualmachine creating or updating\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
correlation ID: 67c21db5-2743-4fb5-a2e0-d37a965dd5f1
client request ID: unavailable
occurredContributing tests (8)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 3.08%2 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 3 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1f4a465481a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gvns5xqr82fn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"name": "568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"status": "Failed",
"startTime": "2026-11-13T06:13:10.507550785Z",
"endTime": "2026-11-13T06:32:15.73883621Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-9sd6w-4wd6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-9sd6w-hng7m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-9sd6w-4wd6d: Creating: virtualmachine creating or updating\nMachine cluster-np-1-9sd6w-hng7m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
correlation ID: 218fb2af-fc43-4aa1-8295-19449adca4e0
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1f4a465481a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gvns5xqr82fn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"name": "568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"status": "Failed",
"startTime": "2026-11-13T06:13:10.507550785Z",
"endTime": "2026-11-13T06:32:15.73883621Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-9sd6w-4wd6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-9sd6w-hng7m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-9sd6w-4wd6d: Creating: virtualmachine creating or updating\nMachine cluster-np-1-9sd6w-hng7m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
correlation ID: 218fb2af-fc43-4aa1-8295-19449adca4e0
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3b86480f0000>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ll4kkgrplwxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613",
"name": "0d38b567-2f27-471a-84ef-558a5c67e613",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.72317124Z",
"endTime": "2026-11-11T01:32:09.867050757Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-n2m78-xpsvx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-n2m78-xpsvx: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
correlation ID: 16057b98-aa78-45ee-a357-4e003f509084
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3b86480f0000>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ll4kkgrplwxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613",
"name": "0d38b567-2f27-471a-84ef-558a5c67e613",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.72317124Z",
"endTime": "2026-11-11T01:32:09.867050757Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-n2m78-xpsvx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-n2m78-xpsvx: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
correlation ID: 16057b98-aa78-45ee-a357-4e003f509084
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x398924e8c5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rb5c9dvb42gv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84",
"name": "21d8f1e5-67ed-479d-8eac-deaac258db84",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.698291582Z",
"endTime": "2026-11-11T01:32:09.741837947Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-8jqlq-kjgfc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-8jqlq-kjgfc: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
correlation ID: 5eca5994-d282-425a-909e-85f0c80c01f6
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x398924e8c5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rb5c9dvb42gv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84",
"name": "21d8f1e5-67ed-479d-8eac-deaac258db84",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.698291582Z",
"endTime": "2026-11-11T01:32:09.741837947Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-8jqlq-kjgfc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-8jqlq-kjgfc: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
correlation ID: 5eca5994-d282-425a-909e-85f0c80c01f6
client request ID: unavailableContributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 3.08%2 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 2 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-c2rs2z" is viable
Unexpected error:
<*errors.joinError | 0x2d2951054f60>:
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d2950dcc760>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d2950dcc720>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d29504c3180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-c2rs2z" is viable
Unexpected error:
<*errors.joinError | 0x2d2951054f60>:
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d2950dcc760>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d2950dcc720>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d29504c3180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:105]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x3e1fd491ab40>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e1fd3cde060>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd3cde040>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3a44e60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3e1fd4123c80>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd4123c60>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3e07540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:105]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x3e1fd491ab40>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e1fd3cde060>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd3cde040>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3a44e60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3e1fd4123c80>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd4123c60>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3e07540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 3.08%2 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 2 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x38b8c17a6b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8h7cslrzs4d4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"name": "cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"status": "Failed",
"startTime": "2026-11-13T07:22:34.538735812Z",
"endTime": "2026-11-13T07:41:41.602384099Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-lv9kw-7q9mn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-lv9kw-7q9mn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
correlation ID: 9af047e6-a985-412b-a5f1-08a259c865d2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x38b8c17a6b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8h7cslrzs4d4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"name": "cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"status": "Failed",
"startTime": "2026-11-13T07:22:34.538735812Z",
"endTime": "2026-11-13T07:41:41.602384099Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-lv9kw-7q9mn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-lv9kw-7q9mn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
correlation ID: 9af047e6-a985-412b-a5f1-08a259c865d2
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1bcb38aea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-ccvxrx6srd7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"name": "ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"status": "Failed",
"startTime": "2026-11-13T03:38:48.906568999Z",
"endTime": "2026-11-13T03:57:50.268469646Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-x596d-7l8cz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-x596d-7l8cz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
correlation ID: 975c1ede-92c5-42fb-b310-1b83c34bc7da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1bcb38aea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-ccvxrx6srd7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"name": "ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"status": "Failed",
"startTime": "2026-11-13T03:38:48.906568999Z",
"endTime": "2026-11-13T03:57:50.268469646Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-x596d-7l8cz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-x596d-7l8cz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
correlation ID: 975c1ede-92c5-42fb-b310-1b83c34bc7da
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 3.08%2 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x164258d863c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xs6vdbh9ddbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01",
"name": "7558c7b9-327a-43c2-8293-9ac1255cbb01",
"status": "Failed",
"startTime": "2026-11-13T04:30:51.785572468Z",
"endTime": "2026-11-13T04:50:00.485199208Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
correlation ID: e0fc903a-bf7b-4e5b-a7e3-14066f61f36f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x164258d863c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xs6vdbh9ddbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01",
"name": "7558c7b9-327a-43c2-8293-9ac1255cbb01",
"status": "Failed",
"startTime": "2026-11-13T04:30:51.785572468Z",
"endTime": "2026-11-13T04:50:00.485199208Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
correlation ID: e0fc903a-bf7b-4e5b-a7e3-14066f61f36f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0x37b08a783e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-m8jlgbs48ts9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"name": "fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"status": "Failed",
"startTime": "2026-11-08T02:36:57.421300337Z",
"endTime": "2026-11-08T02:56:03.454435501Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
correlation ID: b0decc5d-318d-423b-aa5d-75e7753c16f6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0x37b08a783e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-m8jlgbs48ts9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"name": "fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"status": "Failed",
"startTime": "2026-11-08T02:36:57.421300337Z",
"endTime": "2026-11-08T02:56:03.454435501Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
correlation ID: b0decc5d-318d-423b-aa5d-75e7753c16f6
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...full failure pattern: ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 3.08%2 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 1 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
<*fmt.wrapError | 0x29acc44990c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-disabled-image-registry-hcp-cluster-gwmhk7" in resourcegroup="disabled-image-registry-6jmbtjt72wm7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 49965461-a7c3-410f-b49e-8883df244823",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-tez7gwgoxyw3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
correlation ID: 9da5bde4-a864-47a7-bbec-d7354cb68d5b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
<*fmt.wrapError | 0x29acc44990c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-disabled-image-registry-hcp-cluster-gwmhk7" in resourcegroup="disabled-image-registry-6jmbtjt72wm7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 49965461-a7c3-410f-b49e-8883df244823",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-tez7gwgoxyw3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
correlation ID: 9da5bde4-a864-47a7-bbec-d7354cb68d5b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:143]: failed to create customer resources for cluster "cluster-candidate-4-20-xw84kr"
Unexpected error:
<*fmt.wrapError | 0x2722651c160>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz" in resourcegroup="rg-candidate-4-20-zwlln5grl4vh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 5c19921f-4101-4b07-aab3-b71cf2831d22",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-6xl6wx5rad6lg' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
correlation ID: 3b996925-9aa8-423b-8b22-869e2b8ad84e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:143]: failed to create customer resources for cluster "cluster-candidate-4-20-xw84kr"
Unexpected error:
<*fmt.wrapError | 0x2722651c160>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz" in resourcegroup="rg-candidate-4-20-zwlln5grl4vh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 5c19921f-4101-4b07-aab3-b71cf2831d22",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-6xl6wx5rad6lg' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
correlation ID: 3b996925-9aa8-423b-8b22-869e2b8ad84e
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 70 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2b3cc0e02480>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-ntbnht6v2z5z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: f81e3479-b209-4c6e-aba4-7e18ebafd0a6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2b3cc0e02480>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-ntbnht6v2z5z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: f81e3479-b209-4c6e-aba4-7e18ebafd0a6
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x2e4d7a8d39c0>:
failed to create HCP cluster des-encrypt: failed starting cluster creation "des-encrypt" in resourcegroup="des-encrypt-jtt2vbmjr52z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
correlation ID: 36af9c36-44f3-4b61-9fd9-9d1d6dda7a66
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x2e4d7a8d39c0>:
failed to create HCP cluster des-encrypt: failed starting cluster creation "des-encrypt" in resourcegroup="des-encrypt-jtt2vbmjr52z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
correlation ID: 36af9c36-44f3-4b61-9fd9-9d1d6dda7a66
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x13d76780a900>:
failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-zcw7mckfcbxs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
correlation ID: 65af53cb-dfeb-48dd-a0ae-0cb96b88dc93
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x13d76780a900>:
failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-zcw7mckfcbxs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
correlation ID: 65af53cb-dfeb-48dd-a0ae-0cb96b88dc93
client request ID: unavailable
occurredContributing tests (70)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClu...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 70 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x345936a89300>:
failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-gz55ptdrthl5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
correlation ID: 5078ed5a-a408-4bdb-ab12-939f6b2d08de
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x345936a89300>:
failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-gz55ptdrthl5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
correlation ID: 5078ed5a-a408-4bdb-ab12-939f6b2d08de
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x1342f0a44f80>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-d52pnvslfkxx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: d28b148b-a0dd-42a6-90c0-05ebc5318c9c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x1342f0a44f80>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-d52pnvslfkxx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: d28b148b-a0dd-42a6-90c0-05ebc5318c9c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-95k64p" with version 4.20.45 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xd8dfd2812e0>:
failed to create HCP cluster cluster-zstream-4-20-95k64p: failed starting cluster creation "cluster-zstream-4-20-95k64p" in resourcegroup="rg-zstream-upgrade-4-20-zmsrshjlzqg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
correlation ID: 7b753e11-21cc-4524-9f3d-c18e901193f4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-95k64p" with version 4.20.45 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xd8dfd2812e0>:
failed to create HCP cluster cluster-zstream-4-20-95k64p: failed starting cluster creation "cluster-zstream-4-20-95k64p" in resourcegroup="rg-zstream-upgrade-4-20-zmsrshjlzqg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
correlation ID: 7b753e11-21cc-4524-9f3d-c18e901193f4
client request ID: unavailable
occurredContributing tests (70)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code VMExtensionProvisioningError; detail message VM has reported a user failure when processing extension 'installTools'. Please correct the ...full failure pattern: ERROR CODE: DeploymentFailed; detail code VMExtensionProvisioningError; detail message VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28 [stdout] [stderr] : failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at <url>; provider Microsoft.Azure.Extensions Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 3 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:164]: failed to deploy authorized CIDR VM
Unexpected error:
<*fmt.wrapError | 0x30966ca6aea0>:
failed waiting for deployment "test-vm" in resourcegroup="feature-aggregation-qfjvggxrkqkk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 6df45427-1238-4ab2-891c-02cad5db8d36",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
correlation ID: 4d33796d-6c34-40a7-af77-092588cf96e7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:164]: failed to deploy authorized CIDR VM
Unexpected error:
<*fmt.wrapError | 0x30966ca6aea0>:
failed waiting for deployment "test-vm" in resourcegroup="feature-aggregation-qfjvggxrkqkk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 6df45427-1238-4ab2-891c-02cad5db8d36",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
correlation ID: 4d33796d-6c34-40a7-af77-092588cf96e7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:101]: failed to deploy test VM for fully private with KV verification
Unexpected error:
<*fmt.wrapError | 0x259ff4de4d00>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="full-priv-kv-f9598tqlfc6q" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e1a85dbd-0af5-4f1c-b2fd-7725cc39cb74",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
correlation ID: ede4f526-5bd4-4643-b427-cc65fa41cb78
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:101]: failed to deploy test VM for fully private with KV verification
Unexpected error:
<*fmt.wrapError | 0x259ff4de4d00>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="full-priv-kv-f9598tqlfc6q" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e1a85dbd-0af5-4f1c-b2fd-7725cc39cb74",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
correlation ID: ede4f526-5bd4-4643-b427-cc65fa41cb78
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:92]: failed to deploy test VM
Unexpected error:
<*fmt.wrapError | 0x3985cd6a4d40>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-98tlpt4n2kvs" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: af5f1f5c-a0eb-4321-aa22-e99f3ab0afc8",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
correlation ID: b42cc832-94c9-4550-b4e8-74a1692fb2fa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:92]: failed to deploy test VM
Unexpected error:
<*fmt.wrapError | 0x3985cd6a4d40>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-98tlpt4n2kvs" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: af5f1f5c-a0eb-4321-aa22-e99f3ab0afc8",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
correlation ID: b42cc832-94c9-4550-b4e8-74a1692fb2fa
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 2 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj"
Unexpected error:
<*fmt.wrapError | 0x9ce8fef8480>:
failed to create NodePool np-hw65wj: failed waiting for nodepool="np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj" in resourcegroup="rg-candidate-4-22-njg7rgmmg8l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d",
"name": "63497251-36b4-416b-a60b-757bca5b1a6d",
"status": "Failed",
"startTime": "2026-11-13T03:59:57.353246771Z",
"endTime": "2026-11-13T04:19:05.626847985Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-hw65wj-h7fm8-hlfd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-hw65wj-h7fm8-hlfd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
correlation ID: f5bd758f-09c6-4157-8de6-ff700a1dc700
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj"
Unexpected error:
<*fmt.wrapError | 0x9ce8fef8480>:
failed to create NodePool np-hw65wj: failed waiting for nodepool="np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj" in resourcegroup="rg-candidate-4-22-njg7rgmmg8l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d",
"name": "63497251-36b4-416b-a60b-757bca5b1a6d",
"status": "Failed",
"startTime": "2026-11-13T03:59:57.353246771Z",
"endTime": "2026-11-13T04:19:05.626847985Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-hw65wj-h7fm8-hlfd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-hw65wj-h7fm8-hlfd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
correlation ID: f5bd758f-09c6-4157-8de6-ff700a1dc700
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5"
Unexpected error:
<*fmt.wrapError | 0x3acf2e617ec0>:
failed to create NodePool np-4smnl5: failed waiting for nodepool="np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5" in resourcegroup="rg-candidate-4-23-bqlkqphqlrhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"name": "d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"status": "Failed",
"startTime": "2026-11-13T04:02:01.972677601Z",
"endTime": "2026-11-13T04:21:05.56135744Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-4smnl5-nm98c-k5msg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-4smnl5-nm98c-k5msg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
correlation ID: fc7ff880-2948-4807-a4fc-dcd90799df98
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5"
Unexpected error:
<*fmt.wrapError | 0x3acf2e617ec0>:
failed to create NodePool np-4smnl5: failed waiting for nodepool="np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5" in resourcegroup="rg-candidate-4-23-bqlkqphqlrhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"name": "d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"status": "Failed",
"startTime": "2026-11-13T04:02:01.972677601Z",
"endTime": "2026-11-13T04:21:05.56135744Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-4smnl5-nm98c-k5msg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-4smnl5-nm98c-k5msg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
correlation ID: fc7ff880-2948-4807-a4fc-dcd90799df98
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpe...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 2 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e6d92904060>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: f5c2b31d-3614-44bc-8d24-b7b0bc55b26b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e6d92904060>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: f5c2b31d-3614-44bc-8d24-b7b0bc55b26b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2d186fe117a0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: 43414ce7-64c3-4688-88ca-39da1413fc90
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2d186fe117a0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: 43414ce7-64c3-4688-88ca-39da1413fc90
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/h...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 2 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x1eb25ebe180>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 161d28ed-0b64-4007-9411-b5625c5cd6bb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x1eb25ebe180>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 161d28ed-0b64-4007-9411-b5625c5cd6bb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x391f95172ea0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 363bf3d3-c949-4f22-90ee-7ad33b4e32aa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x391f95172ea0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 363bf3d3-c949-4f22-90ee-7ad33b4e32aa
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
fetching manifest <url>full failure pattern: fetching manifest <url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 2 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x320cc6eca0e0>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x320cc6882108>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x320cc655b860>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x320cc6eca060>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x320cc60b1250>{s: "HTTP 504"},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x320cc6eca0e0>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x320cc6882108>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x320cc655b860>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x320cc6eca060>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x320cc60b1250>{s: "HTTP 504"},
},
],
},
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x18ddcdc0fa40>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x18ddcdc24eb8>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x18ddcd102320>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x18ddcdc0fa00>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x18ddcf55f050>{s: "HTTP 504"},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x18ddcdc0fa40>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x18ddcdc24eb8>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x18ddcd102320>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x18ddcdc0fa00>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x18ddcf55f050>{s: "HTTP 504"},
},
],
},
}Contributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: AzureMachine status.initialization.provisioned is false; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.46
Unexpected error:
<*fmt.wrapError | 0x298ff8376680>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-hf5grb" in resourcegroup="rg-np-skip-minor-hf5grb-482lzd4v57gn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05",
"name": "a5613375-9443-4d68-9e17-0ff7a2c93d05",
"status": "Failed",
"startTime": "2026-11-13T07:18:53.583036999Z",
"endTime": "2026-11-13T07:38:01.651580211Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-nps-420-422-5r77b-9fj4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-nps-420-422-5r77b-zmkv2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-nps-420-422-5r77b-9fj4g: NotReady: AzureMachine status.initialization.provisioned is false\nMachine cluster-nps-420-422-5r77b-zmkv2: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
correlation ID: 9bc3ecfd-be4c-41ac-bac1-303b965f5bd5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.46
Unexpected error:
<*fmt.wrapError | 0x298ff8376680>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-hf5grb" in resourcegroup="rg-np-skip-minor-hf5grb-482lzd4v57gn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05",
"name": "a5613375-9443-4d68-9e17-0ff7a2c93d05",
"status": "Failed",
"startTime": "2026-11-13T07:18:53.583036999Z",
"endTime": "2026-11-13T07:38:01.651580211Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-nps-420-422-5r77b-9fj4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-nps-420-422-5r77b-zmkv2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-nps-420-422-5r77b-9fj4g: NotReady: AzureMachine status.initialization.provisioned is false\nMachine cluster-nps-420-422-5r77b-zmkv2: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
correlation ID: 9bc3ecfd-be4c-41ac-bac1-303b965f5bd5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpe...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0x39262adcd200>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 90e0405d-cf96-4f5e-8535-4b7476f99e20
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0x39262adcd200>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 90e0405d-cf96-4f5e-8535-4b7476f99e20
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/h...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xfabd067a0f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
correlation ID: 6767984a-deb2-4f60-8891-913076f601ac
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xfabd067a0f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
correlation ID: 6767984a-deb2-4f60-8891-913076f601ac
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x691f5c08960>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7ndhwr7f8prm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761",
"name": "3852afe7-3cef-47a2-b8aa-4d84a9736761",
"status": "Failed",
"startTime": "2026-11-13T06:16:07.058086055Z",
"endTime": "2026-11-13T06:35:15.725926802Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-g2fdm-7l2bn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-cilium-np-g2fdm-7l2bn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
correlation ID: f6bc5e83-47cf-48b3-a570-66ef10fd1d63
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x691f5c08960>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7ndhwr7f8prm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761",
"name": "3852afe7-3cef-47a2-b8aa-4d84a9736761",
"status": "Failed",
"startTime": "2026-11-13T06:16:07.058086055Z",
"endTime": "2026-11-13T06:35:15.725926802Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-g2fdm-7l2bn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-cilium-np-g2fdm-7l2bn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
correlation ID: f6bc5e83-47cf-48b3-a570-66ef10fd1d63
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:161]: failed to create node pool "np-1" for fully private cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x26ec0f50ace0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fully-private" in resourcegroup="fully-private-7bcdq64qvfpx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078",
"name": "b11c3892-1773-4bc2-900f-66570c5c0078",
"status": "Failed",
"startTime": "2026-11-11T01:15:23.863928224Z",
"endTime": "2026-11-11T01:34:29.732010151Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9h4sc-vdszr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9h4sc-vdszr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
correlation ID: e0a628bb-254f-4199-8b05-5ab8214ab941
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:161]: failed to create node pool "np-1" for fully private cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x26ec0f50ace0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fully-private" in resourcegroup="fully-private-7bcdq64qvfpx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078",
"name": "b11c3892-1773-4bc2-900f-66570c5c0078",
"status": "Failed",
"startTime": "2026-11-11T01:15:23.863928224Z",
"endTime": "2026-11-11T01:34:29.732010151Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9h4sc-vdszr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9h4sc-vdszr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
correlation ID: e0a628bb-254f-4199-8b05-5ab8214ab941
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33f63d810100>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-7xq4wmsrcmrp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"name": "b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"status": "Failed",
"startTime": "2026-11-11T01:23:45.973097536Z",
"endTime": "2026-11-11T01:42:49.787552422Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-8b65r-bz5nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-8b65r-bz5nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
correlation ID: 5b1f6b6f-4c84-444d-b026-e149bc59ef51
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33f63d810100>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-7xq4wmsrcmrp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"name": "b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"status": "Failed",
"startTime": "2026-11-11T01:23:45.973097536Z",
"endTime": "2026-11-11T01:42:49.787552422Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-8b65r-bz5nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-8b65r-bz5nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
correlation ID: 5b1f6b6f-4c84-444d-b026-e149bc59ef51
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0xe89ee532e20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-t7ttl6fk8v4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02",
"name": "4691ec6a-977d-4467-bb77-e056805ada02",
"status": "Failed",
"startTime": "2026-11-11T01:27:04.62054198Z",
"endTime": "2026-11-11T01:46:09.929731411Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-dkgxm-8trx6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-dkgxm-8trx6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
correlation ID: ad20e79e-c229-41ea-a8e2-d72265e902c5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0xe89ee532e20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-t7ttl6fk8v4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02",
"name": "4691ec6a-977d-4467-bb77-e056805ada02",
"status": "Failed",
"startTime": "2026-11-11T01:27:04.62054198Z",
"endTime": "2026-11-11T01:46:09.929731411Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-dkgxm-8trx6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-dkgxm-8trx6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
correlation ID: ad20e79e-c229-41ea-a8e2-d72265e902c5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:192]: failed to create node pool "np-1" for fully private cluster with KV "full-priv-kv"
Unexpected error:
<*fmt.wrapError | 0x399db18e1c20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "full-priv-kv" in resourcegroup="full-priv-kv-xts7895q2w9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5",
"name": "6ee03316-feff-4093-9246-bef8a3716fd5",
"status": "Failed",
"startTime": "2026-11-13T06:16:32.006714258Z",
"endTime": "2026-11-13T06:35:35.830707205Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-whczx-4dxn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-whczx-4dxn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
correlation ID: 003842c9-adb0-44c1-b432-98cc60f79fe8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:192]: failed to create node pool "np-1" for fully private cluster with KV "full-priv-kv"
Unexpected error:
<*fmt.wrapError | 0x399db18e1c20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "full-priv-kv" in resourcegroup="full-priv-kv-xts7895q2w9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5",
"name": "6ee03316-feff-4093-9246-bef8a3716fd5",
"status": "Failed",
"startTime": "2026-11-13T06:16:32.006714258Z",
"endTime": "2026-11-13T06:35:35.830707205Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-whczx-4dxn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-whczx-4dxn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
correlation ID: 003842c9-adb0-44c1-b432-98cc60f79fe8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc"
Unexpected error:
<*fmt.wrapError | 0x4de899b99e0>:
failed to create NodePool np-qzz6xc: failed waiting for nodepool="np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc" in resourcegroup="rg-candidate-4-21-qzgwvl6kwrmt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"name": "f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"status": "Failed",
"startTime": "2026-11-13T06:42:31.271421653Z",
"endTime": "2026-11-13T07:01:35.883206031Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-qzz6xc-57t85-pxcdm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-qzz6xc-57t85-pxcdm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
correlation ID: 99f225d1-e26c-43e5-9f87-e0f4765d3804
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc"
Unexpected error:
<*fmt.wrapError | 0x4de899b99e0>:
failed to create NodePool np-qzz6xc: failed waiting for nodepool="np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc" in resourcegroup="rg-candidate-4-21-qzgwvl6kwrmt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"name": "f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"status": "Failed",
"startTime": "2026-11-13T06:42:31.271421653Z",
"endTime": "2026-11-13T07:01:35.883206031Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-qzz6xc-57t85-pxcdm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-qzz6xc-57t85-pxcdm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
correlation ID: 99f225d1-e26c-43e5-9f87-e0f4765d3804
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-<id> to exist Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.46
Unexpected error:
<*fmt.wrapError | 0xbf3c7f945e0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-2klx7r" in resourcegroup="rg-np-version-upgrade-2klx7r-qhhp8zd44mhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"name": "2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"status": "Failed",
"startTime": "2026-11-13T04:32:00.613107854Z",
"endTime": "2026-11-13T04:51:00.48916791Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
correlation ID: 628c311f-b207-4ece-9d6b-da4a010d2a2c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.46
Unexpected error:
<*fmt.wrapError | 0xbf3c7f945e0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-2klx7r" in resourcegroup="rg-np-version-upgrade-2klx7r-qhhp8zd44mhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"name": "2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"status": "Failed",
"startTime": "2026-11-13T04:32:00.613107854Z",
"endTime": "2026-11-13T04:51:00.48916791Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
correlation ID: 628c311f-b207-4ece-9d6b-da4a010d2a2c
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2971806857e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rplp5xfl4lpm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233",
"name": "c08cc337-7979-4e36-800f-25b7ff101233",
"status": "Failed",
"startTime": "2026-11-13T06:56:27.715876172Z",
"endTime": "2026-11-13T07:15:36.395648013Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine cluster-autoscale-az-45l8j-q7vtn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-autoscale-az-45l8j-q7vtn: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
correlation ID: 66603ef9-ab0b-4acc-9e94-e684aebd41a8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2971806857e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rplp5xfl4lpm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233",
"name": "c08cc337-7979-4e36-800f-25b7ff101233",
"status": "Failed",
"startTime": "2026-11-13T06:56:27.715876172Z",
"endTime": "2026-11-13T07:15:36.395648013Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine cluster-autoscale-az-45l8j-q7vtn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-autoscale-az-45l8j-q7vtn: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
correlation ID: 66603ef9-ab0b-4acc-9e94-e684aebd41a8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xc4ddaff8bb8>:
VerifyNodesReady failed: there are not ready nodes: cluster-agg-np-a-dz7h5-wdm8j (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-11-13T06:34:17Z)
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v94sn66nsnfk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5",
"name": "427fbf99-b011-448a-8853-e0e336f802e5",
"status": "Failed",
"startTime": "2026-11-13T06:16:51.858643855Z",
"endTime": "2026-11-13T06:35:55.74861517Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-agg-np-a-dz7h5-wdm8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-agg-np-a-dz7h5-wdm8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
correlation ID: 3d6b880e-0708-43ee-96e6-2869a452519b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xc4ddaff8bb8>:
VerifyNodesReady failed: there are not ready nodes: cluster-agg-np-a-dz7h5-wdm8j (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-11-13T06:34:17Z)
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v94sn66nsnfk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5",
"name": "427fbf99-b011-448a-8853-e0e336f802e5",
"status": "Failed",
"startTime": "2026-11-13T06:16:51.858643855Z",
"endTime": "2026-11-13T06:35:55.74861517Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-agg-np-a-dz7h5-wdm8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-agg-np-a-dz7h5-wdm8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
correlation ID: 3d6b880e-0708-43ee-96e6-2869a452519b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentityfull failure pattern: ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentity Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:158]: failed to create customer resources for patch-tags cluster
Unexpected error:
<*fmt.wrapError | 0x8054d89efc0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-tlq6j2" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20261110T031257Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '8a115947-73d2-4084-928a-c2672e44e2fc', request correlation id '28d599ba-f561-4e3b-af9c-cf080eee62d4'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
correlation ID: 28a0fd91-bacb-4401-9842-5185b0320dbf
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:158]: failed to create customer resources for patch-tags cluster
Unexpected error:
<*fmt.wrapError | 0x8054d89efc0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-tlq6j2" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20261110T031257Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '8a115947-73d2-4084-928a-c2672e44e2fc', request correlation id '28d599ba-f561-4e3b-af9c-cf080eee62d4'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
correlation ID: 28a0fd91-bacb-4401-9842-5185b0320dbf
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed outfull failure pattern: creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x179a7ca89ca0>:
creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out
{
msg: "creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out",
err: <*errors.StatusError | 0x179a7c45ea00>{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "etcdserver: request timed out",
Reason: "",
Details: nil,
Code: 500,
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x179a7ca89ca0>:
creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out
{
msg: "creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out",
err: <*errors.StatusError | 0x179a7c45ea00>{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "etcdserver: request timed out",
Reason: "",
Details: nil,
Code: 500,
},
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "<vm>" (output: "000")full failure pattern: run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "<vm>" (output: "000") Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.54%1 of 65 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.666s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
<*errors.errorString | 0x1502950d1900>:
run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
{
s: "run command \"e2e-runcommand-1794361068429775103\" failed with exit code 28 on VM \"private-ingress-test-vm\" (output: \"000\")",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.666s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
<*errors.errorString | 0x1502950d1900>:
run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
{
s: "run command \"e2e-runcommand-1794361068429775103\" failed with exit code 28 on VM \"private-ingress-test-vm\" (output: \"000\")",
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||