Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 115 | 42.61% (off track) | -52.39pp | -0.42pp | 91.43% (96/105) | -0.28pp | 51.04% (49/96) | -1.72pp |
| INT | 95% - All E2E job runs | 39 | 74.36% (off track) | -20.64pp | +21.98pp | n/a | n/a | 96.67% (29/30) | -3.33pp |
| STG | 95% - All E2E job runs | 29 | 48.28% (off track) | -46.72pp | +8.88pp | n/a | n/a | 73.68% (14/19) | +8.68pp |
| PROD | 95% - All E2E job runs | 50 | 58.00% (off track) | -37.00pp | -6.65pp | n/a | n/a | 63.04% (29/46) | -9.68pp |
Environment: DEV
E2E Jobs
264
Success Rate
37.50%
Provision success
Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.88.89% (208/234)
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.47.60% (99/208)
E2E Jobs (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the number of runs in that DEV-only goal basis.115
Success Rate (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the overall success rate for that DEV-only goal basis.42.61%
Provision success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.91.43% (96/105)
E2E success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.51.04% (49/96)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
Successful runs (post-good + batches)
Provision failures
E2E failures
Other failures
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 22 | 8.33%22 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 95 · Nov 4: 36 · Nov 5: 0 · Nov 6: 0 | INT, PROD, STG | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.009s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3aab715013a0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.009s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3aab715013a0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x94790d6cbd0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x94790d6cbd0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.004s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3e00259049c0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.004s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3e00259049c0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (22) | |||||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s) | unknown | 13 | 4.92%13 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s) | Show trend detailsOct 30: 4 · Oct 31: 0 · Nov 1: 1 · Nov 2: 2 · Nov 3: 3 · Nov 4: 2 · Nov 5: 5 · Nov 6: 0 | INT, PROD, STG | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T12:34:57Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T12:37:40Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T12:32:07Z"}Contributing tests (1)
Affected runs (13) | |||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 10 | 3.79%10 of 264 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 3 · Nov 3: 0 · Nov 4: 2 · Nov 5: 5 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-05T22:15:55Z Ended: 2026-11-05T22:43:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8231168-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttaugf6cchaj141ocmm0r5pftdrl9e9-cluster", pod="router-7885cb989c-pwjw8", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2ttaugf6cchaj141ocmm0r5pftdrl9e9-cluster/router-7885cb989c-pwjw8 has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-05T21:20:05Z Ended: 2026-11-05T21:39:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4800512-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tta01jejhd924tq8r8qpo822o17l59g-cluster", pod="router-75cfb887d6-f8wdj", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tta01jejhd924tq8r8qpo822o17l59g-cluster/router-75cfb887d6-f8wdj has been in a non-ready state for longer than 5 minutes. alert fired 13 time(s) (4 unknown, 9 known) Firing 1: State: Resolved Started: 2026-11-05T19:20:40Z Ended: 2026-11-05T19:26:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5769472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tt8da1gap29svvgp7t5ko03nhujlpk1-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tt8da1gap29svvgp7t5ko03nhujlpk1-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-05T19:20:40Z Ended: 2026-11-05T19:26:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5769472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tt8d9mve19tpknv942rdublbsd09upk-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tt8d9mve19tpknv942rdublbsd09upk-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-05T19:20:41Z Ended: 2026-11-05T19:26:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5769472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tt8cu8nhnlur31t7cliq526aa8pcm9f-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tt8cu8nhnlur31t7cliq526aa8pcm9f-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-05T19:21:42Z Ended: 2026-11-05T19:27:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5769472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tt8d9lb2fl7am5ftgukdc57i075g9vm-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tt8d9lb2fl7am5ftgukdc57i075g9vm-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (10) | |||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | provision | 8 | 3.03%8 of 264 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 30: 3 · Oct 31: 0 · Nov 1: 0 · Nov 2: 3 · Nov 3: 1 · Nov 4: 0 · Nov 5: 4 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-05T17:42:48.260Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-05T17:42:49.668Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1?api-version=2021-04-01"
time=2026-11-05T17:42:53.116Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1?api-version=2021-04-01"
time=2026-11-05T17:42:57.256Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1?api-version=2021-04-01"
time=2026-11-05T17:43:05.970Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1?api-version=2021-04-01"
time=2026-11-05T17:43:22.362Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1?api-version=2021-04-01"
time=2026-11-05T17:43:56.836Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1?api-version=2021-04-01"
time=2026-11-05T17:43:56.836Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-05T17:43:57.752Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3895808-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1
time=2026-11-05T17:46:28.224Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/71fbd9d5bebd01c79bb0d3e86f4ee11285f08c571470d0c2fdaa62c72c9d44f1/operationStatuses/08584077062481369043\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e7185659-cac3-4384-be0b-3889702b156b\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/cluster-m7ilm6mu7jfso\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e7185659-cac3-4384-be0b-3889702b156b\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j3895808-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j3895808-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-05T17:48:28.226Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-05T17:48:28.997Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0?api-version=2021-04-01"
time=2026-11-05T17:48:31.142Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0?api-version=2021-04-01"
time=2026-11-05T17:48:35.563Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0?api-version=2021-04-01"
time=2026-11-05T17:48:43.760Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0?api-version=2021-04-01"
time=2026-11-05T17:49:01.141Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0?api-version=2021-04-01"
time=2026-11-05T17:49:35.327Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0?api-version=2021-04-01"
time=2026-11-05T17:49:35.328Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-05T17:49:36.423Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3895808-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fdf068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0
time=2026-11-05T17:51:06.863Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3895808-svc/providers/Microsoft.Resources/deployments/df068460a68d4b935eaf74e9a1ab7521024f2c51943c3799c98654cfd87119c0/operationStatuses/08584077059096319520\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 3e6bbb55-25b8-4dcc-9d2e-8029ae71b9e1\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-05T15:59:28.605Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-05T15:59:30.108Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48?api-version=2021-04-01"
time=2026-11-05T15:59:32.211Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48?api-version=2021-04-01"
time=2026-11-05T15:59:36.426Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48?api-version=2021-04-01"
time=2026-11-05T15:59:45.647Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48?api-version=2021-04-01"
time=2026-11-05T16:00:02.767Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48?api-version=2021-04-01"
time=2026-11-05T16:00:36.774Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48?api-version=2021-04-01"
time=2026-11-05T16:00:36.774Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-05T16:00:38.159Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0819712-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48
time=2026-11-05T16:03:08.714Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/f7c27b1347f308dfc1e9f42077da9ed8516295721ee1aae53a663a2b06972c48/operationStatuses/08584077124480583504\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: bb30419a-3b9c-4c2b-b54b-4d7a27fb93e0\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/cluster-yih57pgnbbpnm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: bb30419a-3b9c-4c2b-b54b-4d7a27fb93e0\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0819712-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j0819712-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-05T16:05:08.716Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-05T16:05:09.571Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f?api-version=2021-04-01"
time=2026-11-05T16:05:11.851Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f?api-version=2021-04-01"
time=2026-11-05T16:05:15.988Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f?api-version=2021-04-01"
time=2026-11-05T16:05:24.489Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f?api-version=2021-04-01"
time=2026-11-05T16:05:41.051Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f?api-version=2021-04-01"
time=2026-11-05T16:06:13.776Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f?api-version=2021-04-01"
time=2026-11-05T16:06:13.777Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-05T16:06:14.841Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0819712-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f
time=2026-11-05T16:07:45.756Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0819712-svc/providers/Microsoft.Resources/deployments/67d94cc3282fa1734c603d19e1e7f7580dd75d386b8cdd05f027fa6ffeff701f/operationStatuses/08584077121112099538\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: d1709b2b-c104-4ce7-b036-10d8b4f2b6ec\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-05T12:02:08.708Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-05T12:02:10.082Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7?api-version=2021-04-01"
time=2026-11-05T12:02:13.332Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7?api-version=2021-04-01"
time=2026-11-05T12:02:17.485Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7?api-version=2021-04-01"
time=2026-11-05T12:02:25.846Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7?api-version=2021-04-01"
time=2026-11-05T12:02:42.613Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7?api-version=2021-04-01"
time=2026-11-05T12:03:16.335Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7?api-version=2021-04-01"
time=2026-11-05T12:03:16.335Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-05T12:03:17.412Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6468736-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7
time=2026-11-05T12:05:17.824Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/b92935357b8bc9260cd4763a8abd7b5903a9276d8e1edfff4f178d211c5088e7/operationStatuses/08584077266886241101\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 3262c13e-343e-479d-80b2-42363133fb19\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/cluster-ntr36r4fqjfpu\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 3262c13e-343e-479d-80b2-42363133fb19\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j6468736-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j6468736-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-05T12:07:17.828Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-05T12:07:18.888Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c?api-version=2021-04-01"
time=2026-11-05T12:07:21.092Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c?api-version=2021-04-01"
time=2026-11-05T12:07:25.212Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c?api-version=2021-04-01"
time=2026-11-05T12:07:33.951Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c?api-version=2021-04-01"
time=2026-11-05T12:07:51.331Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c?api-version=2021-04-01"
time=2026-11-05T12:08:25.485Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c?api-version=2021-04-01"
time=2026-11-05T12:08:25.485Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-05T12:08:26.563Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6468736-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Facb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c
time=2026-11-05T12:09:56.924Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6468736-svc/providers/Microsoft.Resources/deployments/acb500f6da0fa28f260f75b9c4d1d1f89fa87979b424977bd1e6b5c8b8c00a2c/operationStatuses/08584077263794606764\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: d51892df-1a96-410d-aa98-a42796ca024d\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| |||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 6 | 2.27%6 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 3 · Nov 3: 1 · Nov 4: 1 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-84g7z47ns2hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xc2c77c949c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-84g7z47ns2hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7187584-mgmt-1/ocm-arohcpci01-2tt7gcihotni4lc75pk8s6lj51h4ri37-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-84g7z47ns2hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7187584-mgmt-1/ocm-arohcpci01-2tt7gcihotni4lc75pk8s6lj51h4ri37-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0xc2c77e78970>{
s: "dev/canadacentral/ci01-j7187584-mgmt-1/ocm-arohcpci01-2tt7gcihotni4lc75pk8s6lj51h4ri37-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-84g7z47ns2hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xc2c77c949c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-84g7z47ns2hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7187584-mgmt-1/ocm-arohcpci01-2tt7gcihotni4lc75pk8s6lj51h4ri37-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-84g7z47ns2hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7187584-mgmt-1/ocm-arohcpci01-2tt7gcihotni4lc75pk8s6lj51h4ri37-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0xc2c77e78970>{
s: "dev/canadacentral/ci01-j7187584-mgmt-1/ocm-arohcpci01-2tt7gcihotni4lc75pk8s6lj51h4ri37-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-n94shjvl5m6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2774eb980000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-n94shjvl5m6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j9706624-mgmt-1/ocm-arohcpci01-2tshhjm5rq9tnko5r5ccuctqbn5hes2t-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-n94shjvl5m6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j9706624-mgmt-1/ocm-arohcpci01-2tshhjm5rq9tnko5r5ccuctqbn5hes2t-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*errors.errorString | 0x2774eb602240>{
s: "dev/centralus/ci01-j9706624-mgmt-1/ocm-arohcpci01-2tshhjm5rq9tnko5r5ccuctqbn5hes2t-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-n94shjvl5m6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2774eb980000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-n94shjvl5m6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j9706624-mgmt-1/ocm-arohcpci01-2tshhjm5rq9tnko5r5ccuctqbn5hes2t-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-n94shjvl5m6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j9706624-mgmt-1/ocm-arohcpci01-2tshhjm5rq9tnko5r5ccuctqbn5hes2t-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*errors.errorString | 0x2774eb602240>{
s: "dev/centralus/ci01-j9706624-mgmt-1/ocm-arohcpci01-2tshhjm5rq9tnko5r5ccuctqbn5hes2t-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x32bf764485a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
err: <*verifiers.pollObservationError | 0x32bf76448560>{
observation: <*errors.errorString | 0x32bf75b33060>{
s: "dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
},
interruption: <*fmt.wrapError | 0x32bf76448540>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x32bf74bea140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x32bf76448520>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x32bf76f2bb30>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x32bf764485a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
err: <*verifiers.pollObservationError | 0x32bf76448560>{
observation: <*errors.errorString | 0x32bf75b33060>{
s: "dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
},
interruption: <*fmt.wrapError | 0x32bf76448540>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x32bf74bea140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x32bf76448520>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x32bf76f2bb30>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (6) | |||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 5 | 1.89%5 of 264 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 3 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-8kxk77"
Unexpected error:
<*fmt.wrapError | 0x29678d132c60>:
DNS for route host agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x29678daf2900>{
msg: "DNS for agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-8kxk77"
Unexpected error:
<*fmt.wrapError | 0x29678d132c60>:
DNS for route host agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x29678daf2900>{
msg: "DNS for agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dphpq.apps.aro.cluster.s4kf7.j6061312.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-28zc46"
Unexpected error:
<*fmt.wrapError | 0x31e44d2b0100>:
DNS for route host agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x31e44d9d9860>{
msg: "DNS for agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-28zc46"
Unexpected error:
<*fmt.wrapError | 0x31e44d2b0100>:
DNS for route host agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x31e44d9d9860>{
msg: "DNS for agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8klb7.apps.aro.cluster.fgzdx.j3487104.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-h97r7s"
Unexpected error:
<*fmt.wrapError | 0xf6f694a44e0>:
DNS for route host agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0xf6f69750100>{
msg: "DNS for agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-h97r7s"
Unexpected error:
<*fmt.wrapError | 0xf6f694a44e0>:
DNS for route host agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0xf6f69750100>{
msg: "DNS for agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wf5t6.apps.aro.cluster.kcgb5.j9706624.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (4)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...full failure pattern: ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDBSignal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | provision | 5 | 1.89%5 of 264 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsOct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 0 · Nov 4: 2 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-05T20:26:55.437Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-05T20:26:56.930Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f?api-version=2021-04-01"
time=2026-11-05T20:26:59.306Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f?api-version=2021-04-01"
time=2026-11-05T20:27:03.672Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f?api-version=2021-04-01"
time=2026-11-05T20:27:12.481Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f?api-version=2021-04-01"
time=2026-11-05T20:27:29.317Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f?api-version=2021-04-01"
time=2026-11-05T20:28:02.038Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f?api-version=2021-04-01"
time=2026-11-05T20:28:02.038Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-05T20:28:02.835Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3632256%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fdefc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f
time=2026-11-05T20:34:39.410Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/defc77bd68adc8477fd7cb2a91f96b10aa0a3be1972d667ad59de2a7534a396f/operationStatuses/08584076964031022218\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 365e19d5-a4b3-481b-bed6-76c5a8853ec8\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3632256/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 365e19d5-a4b3-481b-bed6-76c5a8853ec8\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3632256/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j3632256\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 0f2a8296-0c2e-4f08-9175-c65826effe4a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 679856c6-32c2-405a-ac0b-785ffbfbf879, Request URI: /apps/769daae8-c2aa-4372-89e4-b35316eefb4f/services/6e578474-148e-4126-8582-33da74cd7b73/partitions/bed2c846-2a06-479c-8664-aa5251e89616/replicas/134376072626305227s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-05T20:33:42.3146237Z, RequestEndTime: 2026-11-05T20:33:47.4084704Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-05T20:32:50.8700803Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.385,\\\\\\\"memory\\\\\\\":25154004.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0203,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":138},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-05T20:33:00.8724889Z\\\\\\\",\\\\\\\"cpu\\\\\\\":13.667,\\\\\\\"memory\\\\\\\":25150136.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0651,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":138},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-05T20:33:10.8762502Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.290,\\\\\\\"memory\\\\\\\":25156484.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0476,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":138},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-05T20:33:20.8919959Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.521,\\\\\\\"memory\\\\\\\":25159088.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1088,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":138},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-05T20:33:30.9077718Z\\\\\\\",\\\\\\\"cpu\\\\\\\":3.899,\\\\\\\"memory\\\\\\\":25148500.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0638,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":138},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-05T20:33:40.9083820Z\\\\\\\",\\\\\\\"cpu\\\\\\\":5.863,\\\\\\\"memory\\\\\\\":25212784.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0604,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":138}]}\\\\r\\\\nRequestStart: 2026-11-05T20:33:42.3146237Z; ResponseTime: 2026-11-05T20:33:47.4084704Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be156.documents.azure.com:11010/apps/769daae8-c2aa-4372-89e4-b35316eefb4f/services/6e578474-148e-4126-8582-33da74cd7b73/partitions/bed2c846-2a06-479c-8664-aa5251e89616/replicas/134376072626305227s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5085.183, ActivityId: 679856c6-32c2-405a-ac0b-785ffbfbf879, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11010 | status: Connected | lkt: 11/5/2026 8:33:47 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-05T20:33:42.3146237Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0025},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-05T20:33:42.3146262Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0105},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-05T20:33:42.3146367Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1449},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-05T20:33:42.3147816Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5086.2527},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-05T20:33:47.4010343Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1243},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-05T20:33:47.4011586Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-05T20:31:29.0652717Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-05T20:31:29.0652717Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-05T20:31:29.0964806Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-04T15:59:21.561Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-04T15:59:23.392Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09?api-version=2021-04-01"
time=2026-11-04T15:59:25.492Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09?api-version=2021-04-01"
time=2026-11-04T15:59:29.881Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09?api-version=2021-04-01"
time=2026-11-04T15:59:38.585Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09?api-version=2021-04-01"
time=2026-11-04T15:59:55.547Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09?api-version=2021-04-01"
time=2026-11-04T16:00:27.758Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09?api-version=2021-04-01"
time=2026-11-04T16:00:27.758Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-04T16:00:29.061Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9968128%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09
time=2026-11-04T16:06:59.999Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/9fafee5cdb86900c36d22eb64ed0504fff8ebfd191798921dc1bdecbaf59cd09/operationStatuses/08584077988573663750\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2f28f4cc-54f9-4fdd-990c-7af972ba2e60\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9968128/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2f28f4cc-54f9-4fdd-990c-7af972ba2e60\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9968128/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j9968128\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: c88263bf-08ba-4810-bd03-e9ae0421d4aa, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 7c7ce5b3-1358-4916-a472-c363e7b18b58, Request URI: /apps/5e3d3626-be23-4fe8-b779-c0df213a9166/services/ff0e9540-41d9-4151-a738-d0ac7722ef80/partitions/e3b1f910-67ba-46d2-bb3f-50ae4a840257/replicas/134382275673325986s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-04T16:06:15.5544599Z, RequestEndTime: 2026-11-04T16:06:20.5543967Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T16:05:21.3433394Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.094,\\\\\\\"memory\\\\\\\":25297476.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0569,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":144},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T16:05:31.3588865Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.808,\\\\\\\"memory\\\\\\\":25283784.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0672,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":144},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T16:05:41.3595885Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.250,\\\\\\\"memory\\\\\\\":25291740.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.061,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":144},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T16:05:51.3606338Z\\\\\\\",\\\\\\\"cpu\\\\\\\":6.113,\\\\\\\"memory\\\\\\\":25231092.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0549,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":144},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T16:06:01.3620551Z\\\\\\\",\\\\\\\"cpu\\\\\\\":11.445,\\\\\\\"memory\\\\\\\":25198304.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0591,\\\\\\\"availableThreads\\\\\\\":32763,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":144},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T16:06:11.3669640Z\\\\\\\",\\\\\\\"cpu\\\\\\\":9.887,\\\\\\\"memory\\\\\\\":25175260.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.2003,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":144}]}\\\\r\\\\nRequestStart: 2026-11-04T16:06:15.5544599Z; ResponseTime: 2026-11-04T16:06:20.5543967Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be187.documents.azure.com:11614/apps/5e3d3626-be23-4fe8-b779-c0df213a9166/services/ff0e9540-41d9-4151-a738-d0ac7722ef80/partitions/e3b1f910-67ba-46d2-bb3f-50ae4a840257/replicas/134382275673325986s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 4991.873, ActivityId: 7c7ce5b3-1358-4916-a472-c363e7b18b58, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11614 | status: Connected | lkt: 11/4/2026 4:06:20 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T16:06:15.5544599Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0033},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T16:06:15.5544632Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0105},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T16:06:15.5544737Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1259},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T16:06:15.5545996Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 4992.4666},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T16:06:20.5470662Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1512},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T16:06:20.5472174Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-04T16:04:01.5537155Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-04T16:04:01.5537155Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-04T16:04:07.5422583Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-04T07:55:39.111Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-04T07:55:40.753Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf?api-version=2021-04-01"
time=2026-11-04T07:55:43.001Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf?api-version=2021-04-01"
time=2026-11-04T07:55:47.118Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf?api-version=2021-04-01"
time=2026-11-04T07:55:55.675Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf?api-version=2021-04-01"
time=2026-11-04T07:56:12.556Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf?api-version=2021-04-01"
time=2026-11-04T07:56:45.536Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf?api-version=2021-04-01"
time=2026-11-04T07:56:45.537Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-04T07:56:46.469Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8083200%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf
time=2026-11-04T08:03:18.228Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/5aafa5caa516d6a488380f1e9e613c8c5e6994912734a9be5fff0783d98555cf/operationStatuses/08584078278795150663\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 4a0bfaf3-3512-4996-95eb-8025cfc2cb2c\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8083200/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 4a0bfaf3-3512-4996-95eb-8025cfc2cb2c\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8083200/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j8083200\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 50d47414-48f2-4769-9b2c-0281b60518aa, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: ef7b4ec1-3312-45de-9f1b-79a3c7ae7b7d, Request URI: /apps/627a34ea-8f68-44e1-ac56-3f39687b6544/services/a1d2d12a-db06-463b-ad28-7a5aaa78b59d/partitions/951182c0-69bd-4ad5-a5c9-65fea0e4424c/replicas/134380627980529554s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-04T08:02:27.3447617Z, RequestEndTime: 2026-11-04T08:02:32.9852806Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T08:01:34.9373389Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.371,\\\\\\\"memory\\\\\\\":25932644.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.071,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":23},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T08:01:44.9528598Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.371,\\\\\\\"memory\\\\\\\":25931368.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0559,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":23},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T08:01:54.9683725Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.833,\\\\\\\"memory\\\\\\\":25908272.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0686,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":23},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T08:02:04.9685651Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.406,\\\\\\\"memory\\\\\\\":25920356.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.082,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":23},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T08:02:14.9693184Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.684,\\\\\\\"memory\\\\\\\":25925008.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0545,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":23},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-04T08:02:24.9697974Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.722,\\\\\\\"memory\\\\\\\":25930752.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0681,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":23}]}\\\\r\\\\nRequestStart: 2026-11-04T08:02:27.3447617Z; ResponseTime: 2026-11-04T08:02:32.9852806Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-centralus1-be424.documents.azure.com:11010/apps/627a34ea-8f68-44e1-ac56-3f39687b6544/services/a1d2d12a-db06-463b-ad28-7a5aaa78b59d/partitions/951182c0-69bd-4ad5-a5c9-65fea0e4424c/replicas/134380627980529554s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5639.774, ActivityId: ef7b4ec1-3312-45de-9f1b-79a3c7ae7b7d, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11010 | status: Connected | lkt: 11/4/2026 8:02:32 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T08:02:27.3447617Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0018},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T08:02:27.3447635Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0049},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T08:02:27.3447684Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0732},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T08:02:27.3448416Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5640.4402},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T08:02:32.9852818Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1222},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-04T08:02:32.9854040Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-04T08:00:14.3410466Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-04T08:00:14.3410466Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-04T08:00:14.3722988Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":871,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 5 | 1.89%5 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 4 · Nov 3: 0 · Nov 4: 0 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xgl5c7tbl2wj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xdb38aad0180>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xgl5c7tbl2wj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j4114432-mgmt-1/ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster router=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002: expected matching summary after test start and within 8m; summaryUID=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002 pass=500a1e17-34e6-4ff6-b196-e05430e985cf at=2026-11-05T11:10:37Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xgl5c7tbl2wj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j4114432-mgmt-1/ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster router=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002: expected matching summary after test start and within 8m; summaryUID=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002 pass=500a1e17-34e6-4ff6-b196-e05430e985cf at=2026-11-05T11:10:37Z",
err: <*verifiers.pollObservationError | 0xdb38aad0160>{
observation: <*errors.errorString | 0xdb38b2f3ad0>{
s: "dev/centralus/ci01-j4114432-mgmt-1/ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster router=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002: expected matching summary after test start and within 8m; summaryUID=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002 pass=500a1e17-34e6-4ff6-b196-e05430e985cf at=2026-11-05T11:10:37Z",
},
interruption: <*fmt.wrapError | 0xdb38aad0140>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4114432-mgmt-1\"; let ns = \"ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster\"; let start = datetime(2026-11-05T10:54:17.2790097Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0xdb38a4b4140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0xdb38aad0120>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4114432-mgmt-1\"; let ns = \"ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster\"; let start = datetime(2026-11-05T10:54:17.2790097Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0xdb38c1901b0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xgl5c7tbl2wj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xdb38aad0180>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xgl5c7tbl2wj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j4114432-mgmt-1/ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster router=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002: expected matching summary after test start and within 8m; summaryUID=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002 pass=500a1e17-34e6-4ff6-b196-e05430e985cf at=2026-11-05T11:10:37Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xgl5c7tbl2wj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j4114432-mgmt-1/ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster router=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002: expected matching summary after test start and within 8m; summaryUID=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002 pass=500a1e17-34e6-4ff6-b196-e05430e985cf at=2026-11-05T11:10:37Z",
err: <*verifiers.pollObservationError | 0xdb38aad0160>{
observation: <*errors.errorString | 0xdb38b2f3ad0>{
s: "dev/centralus/ci01-j4114432-mgmt-1/ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster router=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002: expected matching summary after test start and within 8m; summaryUID=cf78372c-be59-43c1-83f7-80cbd185aa5c node=aks-userswft1-15887986-vmss000002 pass=500a1e17-34e6-4ff6-b196-e05430e985cf at=2026-11-05T11:10:37Z",
},
interruption: <*fmt.wrapError | 0xdb38aad0140>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4114432-mgmt-1\"; let ns = \"ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster\"; let start = datetime(2026-11-05T10:54:17.2790097Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0xdb38a4b4140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0xdb38aad0120>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4114432-mgmt-1\"; let ns = \"ocm-arohcpci01-2tt1574ogb21iar0jfssjrhqltoesolq-cluster\"; let start = datetime(2026-11-05T10:54:17.2790097Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0xdb38c1901b0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x21aad040a880>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
err: <*errors.errorString | 0x21aad0ac5810>{
s: "dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x21aad040a880>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
err: <*errors.errorString | 0x21aad0ac5810>{
s: "dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x29888a0e05e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x29888a0e05c0>{
observation: <*errors.errorString | 0x298889ba30a0>{
s: "dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x29888a0e05a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x298889b76320>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x29888a0e0580>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x29888b4d52f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x29888a0e05e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x29888a0e05c0>{
observation: <*errors.errorString | 0x298889ba30a0>{
s: "dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x29888a0e05a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x298889b76320>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x29888a0e0580>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x29888b4d52f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 4 | 1.52%4 of 264 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 1 · Nov 4: 1 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:113]: failed to create node pool "np-h4cc4p" for cluster "cluster-hs-pre-h4cc4p"
Unexpected error:
<*fmt.wrapError | 0x16852ae0d380>:
failed to create NodePool np-h4cc4p: failed waiting for nodepool="np-h4cc4p" for cluster "cluster-hs-pre-h4cc4p" in resourcegroup="rg-hs-pre--gxq5bfl57qzh" to finish creating: GET https://rp.j0083712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/002dd1e3-1ce0-4378-a240-ec27f6f1072f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/002dd1e3-1ce0-4378-a240-ec27f6f1072f",
"name": "002dd1e3-1ce0-4378-a240-ec27f6f1072f",
"status": "Failed",
"startTime": "2026-11-05T16:29:07.289062628Z",
"endTime": "2026-11-05T16:48:11.352307763Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-h4cc4p-7gqdl-dgsnn: Failed: interfaces failed to create or update. err: failed to get existing resource rg-hs-pre--gxq5bfl57qzh-h4cc4p--managed/cluster-np-h4cc4p-7gqdl-dgsnn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--gxq5bfl57qzh-h4cc4p--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-h4cc4p-7gqdl-dgsnn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd0716542-ef97-46d6-b649-6a2e6d879126' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to htt... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0083712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/002dd1e3-1ce0-4378-a240-ec27f6f1072f
correlation ID: 98e922d1-8659-4de0-b457-d17f0d9e5acb
client request ID: 6aec71c0-a279-43df-8c78-2a0ec5efdfc5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:113]: failed to create node pool "np-h4cc4p" for cluster "cluster-hs-pre-h4cc4p"
Unexpected error:
<*fmt.wrapError | 0x16852ae0d380>:
failed to create NodePool np-h4cc4p: failed waiting for nodepool="np-h4cc4p" for cluster "cluster-hs-pre-h4cc4p" in resourcegroup="rg-hs-pre--gxq5bfl57qzh" to finish creating: GET https://rp.j0083712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/002dd1e3-1ce0-4378-a240-ec27f6f1072f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/002dd1e3-1ce0-4378-a240-ec27f6f1072f",
"name": "002dd1e3-1ce0-4378-a240-ec27f6f1072f",
"status": "Failed",
"startTime": "2026-11-05T16:29:07.289062628Z",
"endTime": "2026-11-05T16:48:11.352307763Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-h4cc4p-7gqdl-dgsnn: Failed: interfaces failed to create or update. err: failed to get existing resource rg-hs-pre--gxq5bfl57qzh-h4cc4p--managed/cluster-np-h4cc4p-7gqdl-dgsnn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--gxq5bfl57qzh-h4cc4p--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-h4cc4p-7gqdl-dgsnn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd0716542-ef97-46d6-b649-6a2e6d879126' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to htt... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0083712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/002dd1e3-1ce0-4378-a240-ec27f6f1072f
correlation ID: 98e922d1-8659-4de0-b457-d17f0d9e5acb
client request ID: 6aec71c0-a279-43df-8c78-2a0ec5efdfc5
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x314a6bf29e00>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-2f9gjfsndpms" to finish creating: GET https://rp.j6526976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/402cb55b-3803-4292-9515-d0bb15b6b684
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/402cb55b-3803-4292-9515-d0bb15b6b684",
"name": "402cb55b-3803-4292-9515-d0bb15b6b684",
"status": "Failed",
"startTime": "2026-11-04T02:44:52.351537047Z",
"endTime": "2026-11-04T03:03:54.090690397Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-whvp8-4xt66: Failed: virtualmachine failed to create or update. err: failed to get existing resource v20260901preview-2f9gjfsndpms--managed/cluster-np-1-whvp8-4xt66 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-2f9gjfsndpms--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whvp8-4xt66\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3871b527-fb1e-4123-b38b-3cb2445a9fc8' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-thr... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6526976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/402cb55b-3803-4292-9515-d0bb15b6b684
correlation ID: 3e09196f-8246-4791-9b85-b646cc5bd1a2
client request ID: b48b3f9d-ed0e-4190-9eed-e3d14f99c1a5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x314a6bf29e00>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-2f9gjfsndpms" to finish creating: GET https://rp.j6526976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/402cb55b-3803-4292-9515-d0bb15b6b684
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/402cb55b-3803-4292-9515-d0bb15b6b684",
"name": "402cb55b-3803-4292-9515-d0bb15b6b684",
"status": "Failed",
"startTime": "2026-11-04T02:44:52.351537047Z",
"endTime": "2026-11-04T03:03:54.090690397Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-whvp8-4xt66: Failed: virtualmachine failed to create or update. err: failed to get existing resource v20260901preview-2f9gjfsndpms--managed/cluster-np-1-whvp8-4xt66 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-2f9gjfsndpms--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whvp8-4xt66\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3871b527-fb1e-4123-b38b-3cb2445a9fc8' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-thr... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6526976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/402cb55b-3803-4292-9515-d0bb15b6b684
correlation ID: 3e09196f-8246-4791-9b85-b646cc5bd1a2
client request ID: b48b3f9d-ed0e-4190-9eed-e3d14f99c1a5
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x2869bca80fa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-fdch8wr5ppfb" to finish creating: GET https://rp.j1207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02f1289a-a32f-4e10-9bf0-64aada97b845
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02f1289a-a32f-4e10-9bf0-64aada97b845",
"name": "02f1289a-a32f-4e10-9bf0-64aada97b845",
"status": "Failed",
"startTime": "2026-11-03T03:32:05.481289367Z",
"endTime": "2026-11-03T03:51:07.295204913Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-l22tz-8xf2f: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-fdch8wr5ppfb--managed/cluster-np-1-l22tz-8xf2f-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-fdch8wr5ppfb--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-l22tz-8xf2f-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b9578979-aab2-4539-a4fd-edc4766d5d36' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02f1289a-a32f-4e10-9bf0-64aada97b845
correlation ID: 60cc26b1-2678-477d-aed7-a2fcac30be1e
client request ID: 23817d8c-ec6d-412c-b99c-b7a71c4606f7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x2869bca80fa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-fdch8wr5ppfb" to finish creating: GET https://rp.j1207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02f1289a-a32f-4e10-9bf0-64aada97b845
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02f1289a-a32f-4e10-9bf0-64aada97b845",
"name": "02f1289a-a32f-4e10-9bf0-64aada97b845",
"status": "Failed",
"startTime": "2026-11-03T03:32:05.481289367Z",
"endTime": "2026-11-03T03:51:07.295204913Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-l22tz-8xf2f: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-fdch8wr5ppfb--managed/cluster-np-1-l22tz-8xf2f-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-fdch8wr5ppfb--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-l22tz-8xf2f-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b9578979-aab2-4539-a4fd-edc4766d5d36' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02f1289a-a32f-4e10-9bf0-64aada97b845
correlation ID: 60cc26b1-2678-477d-aed7-a2fcac30be1e
client request ID: 23817d8c-ec6d-412c-b99c-b7a71c4606f7
occurredContributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 4 | 1.52%4 of 264 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 2 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-54gt2n"
Unexpected error:
<*fmt.wrapError | 0x443db45c2c0>:
DNS for route host agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x443db20a9c0>{
msg: "DNS for agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-54gt2n"
Unexpected error:
<*fmt.wrapError | 0x443db45c2c0>:
DNS for route host agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x443db20a9c0>{
msg: "DNS for agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dt77.apps.aro.cluster.fxs4w.j8581888.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0x1ee4819031c0>:
DNS for route host agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1ee481a4a020>{
msg: "DNS for agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0x1ee4819031c0>:
DNS for route host agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1ee481a4a020>{
msg: "DNS for agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ch65h.apps.aro.cluster.d7pn2.j9708416.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-cg8ss5"
Unexpected error:
<*fmt.wrapError | 0x1836aee12660>:
DNS for route host agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1836aefcf100>{
msg: "DNS for agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-cg8ss5"
Unexpected error:
<*fmt.wrapError | 0x1836aee12660>:
DNS for route host agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1836aefcf100>{
msg: "DNS for agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5phmp.apps.aro.cluster.nxjjc.j2145024.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 4 | 1.52%4 of 264 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 1 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | PROD, STG | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x395c4dff49e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-8qdcwgc2765p" to finish creating: GET https://rp.j2386432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/80125475-07ba-4168-88e8-d47176ce43e0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/80125475-07ba-4168-88e8-d47176ce43e0",
"name": "80125475-07ba-4168-88e8-d47176ce43e0",
"status": "Failed",
"startTime": "2026-11-04T17:08:45.830542498Z",
"endTime": "2026-11-04T17:27:52.105404941Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-55m65-qhkvd: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-8qdcwgc2765p--managed/cluster-np-1-55m65-qhkvd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-8qdcwgc2765p--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-55m65-qhkvd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2386432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/80125475-07ba-4168-88e8-d47176ce43e0
correlation ID: 0fe8e6ee-0842-4a3b-a36f-fab8bda71b64
client request ID: 19c15216-5037-4736-9456-581e01c7e0a2
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x395c4dff49e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-8qdcwgc2765p" to finish creating: GET https://rp.j2386432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/80125475-07ba-4168-88e8-d47176ce43e0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/80125475-07ba-4168-88e8-d47176ce43e0",
"name": "80125475-07ba-4168-88e8-d47176ce43e0",
"status": "Failed",
"startTime": "2026-11-04T17:08:45.830542498Z",
"endTime": "2026-11-04T17:27:52.105404941Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-55m65-qhkvd: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-8qdcwgc2765p--managed/cluster-np-1-55m65-qhkvd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-8qdcwgc2765p--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-55m65-qhkvd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'a76148d4-cb11-4b93-9363-54ba8239b0b1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2386432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/80125475-07ba-4168-88e8-d47176ce43e0
correlation ID: 0fe8e6ee-0842-4a3b-a36f-fab8bda71b64
client request ID: 19c15216-5037-4736-9456-581e01c7e0a2fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xf7c689961e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-xbr2lbp7xpdp" to finish creating: GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"name": "91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"status": "Failed",
"startTime": "2026-11-03T18:00:37.406264058Z",
"endTime": "2026-11-03T18:19:39.541562946Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-wmfl5-9q262: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-wmfl5-txll2: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-xbr2lbp7xpdp--managed/cluster-np-1-wmfl5-txll2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-wmfl5-txll2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\nMachine cluster-np-1-wmfl5-9q262: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
correlation ID: 036f75f3-24b0-4f91-bc59-5746100a9a8a
client request ID: f269ac56-2518-4210-976b-61a6c750a6ce
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xf7c689961e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-xbr2lbp7xpdp" to finish creating: GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"name": "91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"status": "Failed",
"startTime": "2026-11-03T18:00:37.406264058Z",
"endTime": "2026-11-03T18:19:39.541562946Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-wmfl5-9q262: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-wmfl5-txll2: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-xbr2lbp7xpdp--managed/cluster-np-1-wmfl5-txll2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-wmfl5-txll2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\nMachine cluster-np-1-wmfl5-9q262: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
correlation ID: 036f75f3-24b0-4f91-bc59-5746100a9a8a
client request ID: f269ac56-2518-4210-976b-61a6c750a6cefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1554a483a2c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fqqws26gj5wg" to finish creating: GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a",
"name": "99b5366a-e767-4906-ac36-4294137d7f8a",
"status": "Failed",
"startTime": "2026-11-02T18:21:28.76444558Z",
"endTime": "2026-11-02T18:40:30.672040856Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9jhrf-vj7z9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9jhrf-vj7z9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
correlation ID: d32d6140-7634-4f62-8bb9-b4e1bdf31356
client request ID: 313e9b91-f2fc-449a-b3b6-d509cbe4b657
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1554a483a2c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fqqws26gj5wg" to finish creating: GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a",
"name": "99b5366a-e767-4906-ac36-4294137d7f8a",
"status": "Failed",
"startTime": "2026-11-02T18:21:28.76444558Z",
"endTime": "2026-11-02T18:40:30.672040856Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9jhrf-vj7z9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9jhrf-vj7z9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
correlation ID: d32d6140-7634-4f62-8bb9-b4e1bdf31356
client request ID: 313e9b91-f2fc-449a-b3b6-d509cbe4b657Contributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 4 | 1.52%4 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 5 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapError | 0x2abc6c0e2320>:
failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n5htxb65vg4v" to finish creating: GET https://rp.j8231168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3581e5b1-b192-4d11-9094-e9866c6e7afc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3581e5b1-b192-4d11-9094-e9866c6e7afc",
"name": "3581e5b1-b192-4d11-9094-e9866c6e7afc",
"status": "Failed",
"startTime": "2026-11-05T22:03:45.047259267Z",
"endTime": "2026-11-05T22:32:47.454720866Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.41 is Partial (want Completed), started 22m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8231168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3581e5b1-b192-4d11-9094-e9866c6e7afc
correlation ID: 546d55f3-a11b-4c41-bdd2-4ba04e53484a
client request ID: b61e3ba6-9356-433d-b13d-26f260d5e750
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapError | 0x2abc6c0e2320>:
failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n5htxb65vg4v" to finish creating: GET https://rp.j8231168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3581e5b1-b192-4d11-9094-e9866c6e7afc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3581e5b1-b192-4d11-9094-e9866c6e7afc",
"name": "3581e5b1-b192-4d11-9094-e9866c6e7afc",
"status": "Failed",
"startTime": "2026-11-05T22:03:45.047259267Z",
"endTime": "2026-11-05T22:32:47.454720866Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.41 is Partial (want Completed), started 22m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8231168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3581e5b1-b192-4d11-9094-e9866c6e7afc
correlation ID: 546d55f3-a11b-4c41-bdd2-4ba04e53484a
client request ID: b61e3ba6-9356-433d-b13d-26f260d5e750
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:107]: failed to create fully private HCP cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x19548e2c43a0>:
failed to create HCP cluster fully-private: failed waiting for cluster="fully-private" in resourcegroup="fully-private-rwv7db6dfv4l" to finish creating: GET https://rp.j4800512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/419dfecf-443e-48bd-b2f3-e80a2ee1184f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/419dfecf-443e-48bd-b2f3-e80a2ee1184f",
"name": "419dfecf-443e-48bd-b2f3-e80a2ee1184f",
"status": "Failed",
"startTime": "2026-11-05T20:58:45.296512179Z",
"endTime": "2026-11-05T21:27:52.507101176Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 14m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4800512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/419dfecf-443e-48bd-b2f3-e80a2ee1184f
correlation ID: a16fb84d-7dea-45d1-8a53-cf91c8dbfcad
client request ID: 94754845-ea86-4894-8929-6ba7cdb2fce7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:107]: failed to create fully private HCP cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x19548e2c43a0>:
failed to create HCP cluster fully-private: failed waiting for cluster="fully-private" in resourcegroup="fully-private-rwv7db6dfv4l" to finish creating: GET https://rp.j4800512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/419dfecf-443e-48bd-b2f3-e80a2ee1184f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/419dfecf-443e-48bd-b2f3-e80a2ee1184f",
"name": "419dfecf-443e-48bd-b2f3-e80a2ee1184f",
"status": "Failed",
"startTime": "2026-11-05T20:58:45.296512179Z",
"endTime": "2026-11-05T21:27:52.507101176Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 14m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4800512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/419dfecf-443e-48bd-b2f3-e80a2ee1184f
correlation ID: a16fb84d-7dea-45d1-8a53-cf91c8dbfcad
client request ID: 94754845-ea86-4894-8929-6ba7cdb2fce7
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-fbhcnp6zhgpw/cluster-candidate-4-20-bdsmd2 should provision
Unexpected error:
<*fmt.wrapError | 0x2e3457dd22e0>:
failed to create HCP cluster cluster-candidate-4-20-bdsmd2: failed waiting for cluster="cluster-candidate-4-20-bdsmd2" in resourcegroup="rg-candidate-4-20-fbhcnp6zhgpw" to finish creating: GET https://rp.j9802496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6db74093-8ab9-4f87-bfbd-f62354c1f0dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6db74093-8ab9-4f87-bfbd-f62354c1f0dd",
"name": "6db74093-8ab9-4f87-bfbd-f62354c1f0dd",
"status": "Failed",
"startTime": "2026-11-05T18:16:20.157134098Z",
"endTime": "2026-11-05T18:45:25.555033687Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.45 is Partial (want Completed), started 20m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9802496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6db74093-8ab9-4f87-bfbd-f62354c1f0dd
correlation ID: f5a05443-0124-4db5-8d13-ff2bd6167daf
client request ID: 12965a21-b71c-4ccb-98c7-e7ad64872ded
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-fbhcnp6zhgpw/cluster-candidate-4-20-bdsmd2 should provision
Unexpected error:
<*fmt.wrapError | 0x2e3457dd22e0>:
failed to create HCP cluster cluster-candidate-4-20-bdsmd2: failed waiting for cluster="cluster-candidate-4-20-bdsmd2" in resourcegroup="rg-candidate-4-20-fbhcnp6zhgpw" to finish creating: GET https://rp.j9802496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6db74093-8ab9-4f87-bfbd-f62354c1f0dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6db74093-8ab9-4f87-bfbd-f62354c1f0dd",
"name": "6db74093-8ab9-4f87-bfbd-f62354c1f0dd",
"status": "Failed",
"startTime": "2026-11-05T18:16:20.157134098Z",
"endTime": "2026-11-05T18:45:25.555033687Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.45 is Partial (want Completed), started 20m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9802496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6db74093-8ab9-4f87-bfbd-f62354c1f0dd
correlation ID: f5a05443-0124-4db5-8d13-ff2bd6167daf
client request ID: 12965a21-b71c-4ccb-98c7-e7ad64872ded
occurredContributing tests (5)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 4 | 1.52%4 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 0 · Nov 4: 1 · Nov 5: 1 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x551cc2ce440>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x551cbff6a70>{
s: "dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x551cc2ce440>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x551cbff6a70>{
s: "dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-99f8pxb79h2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x35c5986381e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-99f8pxb79h2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j1791360-mgmt-1/ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster router=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001: expected matching summary after test start and within 8m; summaryUID=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001 pass=ae34e9ee-c182-4cfd-98f2-2357e3f20cd0 at=2026-11-04T18:17:44Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-99f8pxb79h2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j1791360-mgmt-1/ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster router=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001: expected matching summary after test start and within 8m; summaryUID=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001 pass=ae34e9ee-c182-4cfd-98f2-2357e3f20cd0 at=2026-11-04T18:17:44Z",
err: <*verifiers.pollObservationError | 0x35c5986381c0>{
observation: <*errors.errorString | 0x35c598efc330>{
s: "dev/centralus/ci01-j1791360-mgmt-1/ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster router=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001: expected matching summary after test start and within 8m; summaryUID=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001 pass=ae34e9ee-c182-4cfd-98f2-2357e3f20cd0 at=2026-11-04T18:17:44Z",
},
interruption: <*fmt.wrapError | 0x35c5986381a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1791360-mgmt-1\"; let ns = \"ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster\"; let start = datetime(2026-11-04T18:02:25.097196Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x35c598a12140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x35c598638180>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1791360-mgmt-1\"; let ns = \"ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster\"; let start = datetime(2026-11-04T18:02:25.097196Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x35c59a66a150>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-99f8pxb79h2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x35c5986381e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-99f8pxb79h2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j1791360-mgmt-1/ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster router=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001: expected matching summary after test start and within 8m; summaryUID=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001 pass=ae34e9ee-c182-4cfd-98f2-2357e3f20cd0 at=2026-11-04T18:17:44Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-99f8pxb79h2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j1791360-mgmt-1/ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster router=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001: expected matching summary after test start and within 8m; summaryUID=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001 pass=ae34e9ee-c182-4cfd-98f2-2357e3f20cd0 at=2026-11-04T18:17:44Z",
err: <*verifiers.pollObservationError | 0x35c5986381c0>{
observation: <*errors.errorString | 0x35c598efc330>{
s: "dev/centralus/ci01-j1791360-mgmt-1/ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster router=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001: expected matching summary after test start and within 8m; summaryUID=46b45587-bf9f-4294-9afa-077db1f9f024 node=aks-userswft1-14844281-vmss000001 pass=ae34e9ee-c182-4cfd-98f2-2357e3f20cd0 at=2026-11-04T18:17:44Z",
},
interruption: <*fmt.wrapError | 0x35c5986381a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1791360-mgmt-1\"; let ns = \"ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster\"; let start = datetime(2026-11-04T18:02:25.097196Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x35c598a12140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x35c598638180>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1791360-mgmt-1\"; let ns = \"ocm-arohcpci01-2tsiargolkqer6ifksngdbdkjn0d87m5-cluster\"; let start = datetime(2026-11-04T18:02:25.097196Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x35c59a66a150>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3a3b4c2860c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
err: <*verifiers.pollObservationError | 0x3a3b4c2860a0>{
observation: <*errors.errorString | 0x3a3b4ae007e0>{
s: "dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
},
interruption: <*fmt.wrapError | 0x3a3b4c286080>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3a3b4b2bc140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3a3b4c286060>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3a3b4cc54210>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3a3b4c2860c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
err: <*verifiers.pollObservationError | 0x3a3b4c2860a0>{
observation: <*errors.errorString | 0x3a3b4ae007e0>{
s: "dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
},
interruption: <*fmt.wrapError | 0x3a3b4c286080>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3a3b4b2bc140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3a3b4c286060>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3a3b4cc54210>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | e2e | 3 | 1.14%3 of 264 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsOct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 1 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x15f68cc94e40>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b85tpn" in resourcegroup="rg-np-version-upgrade-b85tpn-hdfth7g8vbrk" to finish creating: GET https://rp.j2145024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93b33c78-2832-413e-b1a8-2993a7695239
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93b33c78-2832-413e-b1a8-2993a7695239",
"name": "93b33c78-2832-413e-b1a8-2993a7695239",
"status": "Failed",
"startTime": "2026-11-04T06:36:54.179163731Z",
"endTime": "2026-11-04T06:55:56.126205376Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-x44zn-cktww: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-x44zn-cktww: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2145024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93b33c78-2832-413e-b1a8-2993a7695239
correlation ID: 3c95cf7e-236d-492f-8f12-fd44adf27c60
client request ID: 5f148a11-414d-4147-88be-38618590197c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x15f68cc94e40>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b85tpn" in resourcegroup="rg-np-version-upgrade-b85tpn-hdfth7g8vbrk" to finish creating: GET https://rp.j2145024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93b33c78-2832-413e-b1a8-2993a7695239
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93b33c78-2832-413e-b1a8-2993a7695239",
"name": "93b33c78-2832-413e-b1a8-2993a7695239",
"status": "Failed",
"startTime": "2026-11-04T06:36:54.179163731Z",
"endTime": "2026-11-04T06:55:56.126205376Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-x44zn-cktww: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-x44zn-cktww: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2145024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/93b33c78-2832-413e-b1a8-2993a7695239
correlation ID: 3c95cf7e-236d-492f-8f12-fd44adf27c60
client request ID: 5f148a11-414d-4147-88be-38618590197c
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1e4f9fe3a6e0>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-wtdct2" in resourcegroup="rg-np-noedge-wtdct2-l72l4l8jnkw4" to finish creating: GET https://rp.j7446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a570f873-93bb-4987-a1f3-c54b592e3f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a570f873-93bb-4987-a1f3-c54b592e3f26",
"name": "a570f873-93bb-4987-a1f3-c54b592e3f26",
"status": "Failed",
"startTime": "2026-11-04T00:45:35.106345961Z",
"endTime": "2026-11-04T01:04:37.000327364Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-r5mvx-wd8l4: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-r5mvx-wd8l4: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a570f873-93bb-4987-a1f3-c54b592e3f26
correlation ID: a7e025fa-1041-4182-b350-33a5d1edc5a2
client request ID: a287cbdd-cb9e-4d0c-bee2-520e59b73b5e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1e4f9fe3a6e0>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-wtdct2" in resourcegroup="rg-np-noedge-wtdct2-l72l4l8jnkw4" to finish creating: GET https://rp.j7446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a570f873-93bb-4987-a1f3-c54b592e3f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a570f873-93bb-4987-a1f3-c54b592e3f26",
"name": "a570f873-93bb-4987-a1f3-c54b592e3f26",
"status": "Failed",
"startTime": "2026-11-04T00:45:35.106345961Z",
"endTime": "2026-11-04T01:04:37.000327364Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-r5mvx-wd8l4: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-r5mvx-wd8l4: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7446912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a570f873-93bb-4987-a1f3-c54b592e3f26
correlation ID: a7e025fa-1041-4182-b350-33a5d1edc5a2
client request ID: a287cbdd-cb9e-4d0c-bee2-520e59b73b5e
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
<*fmt.wrapError | 0x12b5671d47e0>:
failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-9rkb6kxf5qwc" to finish creating: GET https://rp.j5724928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e669f87-3f8b-40bc-8c94-c82d949437c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e669f87-3f8b-40bc-8c94-c82d949437c3",
"name": "8e669f87-3f8b-40bc-8c94-c82d949437c3",
"status": "Failed",
"startTime": "2026-11-02T03:41:42.541695764Z",
"endTime": "2026-11-02T04:00:42.833093352Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-des-np-7cmlx-vcbqs: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-des-np-7cmlx-vcbqs: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5724928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e669f87-3f8b-40bc-8c94-c82d949437c3
correlation ID: 479195c9-6bea-4ffe-a7ea-a1f0d3f352ff
client request ID: 741ffb9a-bd3c-4268-9b6f-74f56184b2ea
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
<*fmt.wrapError | 0x12b5671d47e0>:
failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-9rkb6kxf5qwc" to finish creating: GET https://rp.j5724928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e669f87-3f8b-40bc-8c94-c82d949437c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e669f87-3f8b-40bc-8c94-c82d949437c3",
"name": "8e669f87-3f8b-40bc-8c94-c82d949437c3",
"status": "Failed",
"startTime": "2026-11-02T03:41:42.541695764Z",
"endTime": "2026-11-02T04:00:42.833093352Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-des-np-7cmlx-vcbqs: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-des-np-7cmlx-vcbqs: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5724928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e669f87-3f8b-40bc-8c94-c82d949437c3
correlation ID: 479195c9-6bea-4ffe-a7ea-a1f0d3f352ff
client request ID: 741ffb9a-bd3c-4268-9b6f-74f56184b2ea
occurredContributing tests (4)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
alert [svc] FrontendPathLatency firedfull failure pattern: alert [svc] FrontendPathLatency fired Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | alert | 3 | 1.14%3 of 264 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 1 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s)
Firing 1:
State: Resolved
Started: 2026-11-04T16:51:38Z
Ended: 2026-11-04T17:16:36Z
Severity: Sev3
Labels: alertname="FrontendPathLatency", cluster="ci01-j9320320-svc", component="frontend", method="post", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential", severity="warning"
Description: More than 5% of frontend requests for post /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential exceeded 1 second over the past 30 minutes (cluster ci01-j9320320-svc).
Firing 2:
State: Resolved
Started: 2026-11-04T17:02:35Z
Ended: 2026-11-04T17:24:36Z
Severity: Sev3
Labels: alertname="FrontendPathLatency", cluster="ci01-j9320320-svc", component="frontend", method="delete", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j9320320-svc).alert fired 1 time(s)
Firing 1:
State: Resolved
Started: 2026-11-03T16:54:20Z
Ended: 2026-11-03T17:03:24Z
Severity: Sev3
Labels: alertname="FrontendPathLatency", cluster="ci01-j5732352-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j5732352-svc).alert fired 1 time(s)
Firing 1:
State: Resolved
Started: 2026-11-02T20:35:41Z
Ended: 2026-11-02T20:57:37Z
Severity: Sev3
Labels: alertname="FrontendPathLatency", cluster="ci01-j8010496-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationstatuses/{operationid}", severity="warning"
Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationstatuses/{operationid} exceeded 1 second over the past 30 minutes (cluster ci01-j8010496-svc).Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
error code: CANCELfull failure pattern: error code: CANCEL Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 1.14%3 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-lgg657"
Unexpected error:
<*errors.joinError | 0x286b5c5375f0>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.002s: PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-5jr5b/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
[pod] e2e-csi-disk-5jr5b/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
[events] e2e-csi-disk-5jr5b Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-5jr5b/e2e-csi-pvc" (reason=Provisioning, count=11, age=29s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-lgg657-7pttx-bg84d (reason=ProvisioningFailed, count=7, age=8m57s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=4, age=9m26s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=2m50s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=2m33s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)
{
errs: [
<*fmt.wrapError | 0x286b5c17c020>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.002s: PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5jr5b/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-5jr5b/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-5jr5b Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5jr5b/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-lgg657-7pttx-bg84d (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=4, age=9m26s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=2m50s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=2m33s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*fmt.wrapError | 0x286b5c8f4380>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.002s: PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5jr5b/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-5jr5b/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-5jr5b Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5jr5b/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-lgg657-7pttx-bg84d (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=4, age=9m26s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=2m50s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=2m33s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*verifiers.pollObservationError | 0x286b5c8d02e0>{
observation: <*errors.errorString | 0x286b5e3027b0>{
s: "PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
},
interruption: <*fmt.wrapError | 0x286b5c8d02c0>{
msg: "failed to get PVC e2e-csi-disk-5jr5b/e2e-csi-pvc: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-lgg657"
Unexpected error:
<*errors.joinError | 0x286b5c5375f0>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.002s: PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-5jr5b/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
[pod] e2e-csi-disk-5jr5b/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
[events] e2e-csi-disk-5jr5b Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-5jr5b/e2e-csi-pvc" (reason=Provisioning, count=11, age=29s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-lgg657-7pttx-bg84d (reason=ProvisioningFailed, count=7, age=8m57s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=4, age=9m26s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=2m50s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=2m33s)
[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)
{
errs: [
<*fmt.wrapError | 0x286b5c17c020>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.002s: PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5jr5b/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-5jr5b/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-5jr5b Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5jr5b/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-lgg657-7pttx-bg84d (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=4, age=9m26s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=2m50s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=2m33s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*fmt.wrapError | 0x286b5c8f4380>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.002s: PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5jr5b/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-5jr5b/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-5jr5b Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5jr5b/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-lgg657-7pttx-bg84d (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=4, age=9m26s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=2m50s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=2m33s)\n[events] e2e-csi-disk-5jr5b PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*verifiers.pollObservationError | 0x286b5c8d02e0>{
observation: <*errors.errorString | 0x286b5e3027b0>{
s: "PVC e2e-csi-disk-5jr5b/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
},
interruption: <*fmt.wrapError | 0x286b5c8d02c0>{
msg: "failed to get PVC e2e-csi-disk-5jr5b/e2e-csi-pvc: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-s9x4rc"
Unexpected error:
<*errors.joinError | 0x394ce8eec258>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-zvhs4/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
[pod] e2e-csi-disk-zvhs4/e2e-csi-pod phase=Pending reason= message=
[events] e2e-csi-disk-zvhs4 Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-zvhs4/e2e-csi-pvc" (reason=Provisioning, count=11, age=29s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=25, age=4m6s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-s9x4rc-nplms-nnx8g (reason=ProvisioningFailed, count=7, age=8m57s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m15s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)
{
errs: [
<*fmt.wrapError | 0x394ce7cfa540>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-zvhs4/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-zvhs4/e2e-csi-pod phase=Pending reason= message=\n[events] e2e-csi-disk-zvhs4 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-zvhs4/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=25, age=4m6s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-s9x4rc-nplms-nnx8g (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m15s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*fmt.wrapError | 0x394ce6d92740>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-zvhs4/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-zvhs4/e2e-csi-pod phase=Pending reason= message=\n[events] e2e-csi-disk-zvhs4 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-zvhs4/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=25, age=4m6s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-s9x4rc-nplms-nnx8g (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m15s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*verifiers.pollObservationError | 0x394ce7cfa360>{
observation: <*errors.errorString | 0x394ce7d2a2c0>{
s: "PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
},
interruption: <*fmt.wrapError | 0x394ce7cfa340>{
msg: "failed to get PVC e2e-csi-disk-zvhs4/e2e-csi-pvc: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-s9x4rc"
Unexpected error:
<*errors.joinError | 0x394ce8eec258>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-zvhs4/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
[pod] e2e-csi-disk-zvhs4/e2e-csi-pod phase=Pending reason= message=
[events] e2e-csi-disk-zvhs4 Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-zvhs4/e2e-csi-pvc" (reason=Provisioning, count=11, age=29s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=25, age=4m6s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-s9x4rc-nplms-nnx8g (reason=ProvisioningFailed, count=7, age=8m57s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m15s)
[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)
{
errs: [
<*fmt.wrapError | 0x394ce7cfa540>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-zvhs4/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-zvhs4/e2e-csi-pod phase=Pending reason= message=\n[events] e2e-csi-disk-zvhs4 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-zvhs4/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=25, age=4m6s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-s9x4rc-nplms-nnx8g (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m15s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*fmt.wrapError | 0x394ce6d92740>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-zvhs4/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-zvhs4/e2e-csi-pod phase=Pending reason= message=\n[events] e2e-csi-disk-zvhs4 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-zvhs4/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=25, age=4m6s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-s9x4rc-nplms-nnx8g (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m15s)\n[events] e2e-csi-disk-zvhs4 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=4m45s)",
err: <*verifiers.pollObservationError | 0x394ce7cfa360>{
observation: <*errors.errorString | 0x394ce7d2a2c0>{
s: "PVC e2e-csi-disk-zvhs4/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
},
interruption: <*fmt.wrapError | 0x394ce7cfa340>{
msg: "failed to get PVC e2e-csi-disk-zvhs4/e2e-csi-pvc: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-zfxr7v"
Unexpected error:
<*errors.joinError | 0x316cc300a798>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-9v857/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-9v857/e2e-csi-pvc phase=Bound volumeName=pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 storageClass=managed-csi
[pod] e2e-csi-disk-9v857/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodReadyToStartContainers status=False reason= message=
[pod] condition type=Initialized status=True reason= message=
[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=PodScheduled status=True reason= message=
[pod] container volume-check ready=false restartCount=0
[pod] waiting reason=ContainerCreating message=
[events] e2e-csi-disk-9v857 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-9v857/e2e-csi-pod to cluster-np-zfxr7v-sppks-m75w5 (reason=Scheduled, count=0, age=24s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m58s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-9v857/e2e-csi-pvc" (reason=Provisioning, count=11, age=29s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-zfxr7v-sppks-m75w5 (reason=ProvisioningFailed, count=7, age=8m57s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m15s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m45s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=1m53s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 (reason=ProvisioningSucceeded, count=1, age=24s)
{
errs: [
<*fmt.wrapError | 0x316cc1246780>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-9v857/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-9v857/e2e-csi-pvc phase=Bound volumeName=pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 storageClass=managed-csi\n[pod] e2e-csi-disk-9v857/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-9v857 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-9v857/e2e-csi-pod to cluster-np-zfxr7v-sppks-m75w5 (reason=Scheduled, count=0, age=24s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m58s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-9v857/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-zfxr7v-sppks-m75w5 (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m15s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m45s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=1m53s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 (reason=ProvisioningSucceeded, count=1, age=24s)",
err: <*fmt.wrapError | 0x316cc1246660>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-9v857/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-9v857/e2e-csi-pvc phase=Bound volumeName=pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 storageClass=managed-csi\n[pod] e2e-csi-disk-9v857/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-9v857 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-9v857/e2e-csi-pod to cluster-np-zfxr7v-sppks-m75w5 (reason=Scheduled, count=0, age=24s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m58s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-9v857/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-zfxr7v-sppks-m75w5 (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m15s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m45s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=1m53s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 (reason=ProvisioningSucceeded, count=1, age=24s)",
err: <*verifiers.pollObservationError | 0x316cc10e24a0>{
observation: <*errors.errorString | 0x316cc2e0c7d0>{
s: "pod e2e-csi-disk-9v857/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
},
interruption: <*fmt.wrapError | 0x316cc10e2440>{
msg: "failed to get PVC e2e-csi-disk-9v857/e2e-csi-pvc: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-zfxr7v"
Unexpected error:
<*errors.joinError | 0x316cc300a798>:
VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-9v857/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
[storageclass] name=managed-csi provisioner=disk.csi.azure.com
[pvc] e2e-csi-disk-9v857/e2e-csi-pvc phase=Bound volumeName=pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 storageClass=managed-csi
[pod] e2e-csi-disk-9v857/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodReadyToStartContainers status=False reason= message=
[pod] condition type=Initialized status=True reason= message=
[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
[pod] condition type=PodScheduled status=True reason= message=
[pod] container volume-check ready=false restartCount=0
[pod] waiting reason=ContainerCreating message=
[events] e2e-csi-disk-9v857 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-9v857/e2e-csi-pod to cluster-np-zfxr7v-sppks-m75w5 (reason=Scheduled, count=0, age=24s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m58s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-9v857/e2e-csi-pvc" (reason=Provisioning, count=11, age=29s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-zfxr7v-sppks-m75w5 (reason=ProvisioningFailed, count=7, age=8m57s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m15s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m45s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=1m53s)
[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 (reason=ProvisioningSucceeded, count=1, age=24s)
{
errs: [
<*fmt.wrapError | 0x316cc1246780>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-9v857/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-9v857/e2e-csi-pvc phase=Bound volumeName=pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 storageClass=managed-csi\n[pod] e2e-csi-disk-9v857/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-9v857 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-9v857/e2e-csi-pod to cluster-np-zfxr7v-sppks-m75w5 (reason=Scheduled, count=0, age=24s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m58s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-9v857/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-zfxr7v-sppks-m75w5 (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m15s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m45s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=1m53s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 (reason=ProvisioningSucceeded, count=1, age=24s)",
err: <*fmt.wrapError | 0x316cc1246660>{
msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: pod e2e-csi-disk-9v857/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-9v857/e2e-csi-pvc phase=Bound volumeName=pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 storageClass=managed-csi\n[pod] e2e-csi-disk-9v857/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod] waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-9v857 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-9v857/e2e-csi-pod to cluster-np-zfxr7v-sppks-m75w5 (reason=Scheduled, count=0, age=24s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=2, age=9m58s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-9v857/e2e-csi-pvc\" (reason=Provisioning, count=11, age=29s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node cluster-np-zfxr7v-sppks-m75w5 (reason=ProvisioningFailed, count=7, age=8m57s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m15s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=2, age=4m45s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=1m53s)\n[events] e2e-csi-disk-9v857 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9e925aac-e58d-473d-b30b-d8ed13d9f090 (reason=ProvisioningSucceeded, count=1, age=24s)",
err: <*verifiers.pollObservationError | 0x316cc10e24a0>{
observation: <*errors.errorString | 0x316cc2e0c7d0>{
s: "pod e2e-csi-disk-9v857/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
},
interruption: <*fmt.wrapError | 0x316cc10e2440>{
msg: "failed to get PVC e2e-csi-disk-9v857/e2e-csi-pvc: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 3 | 1.14%3 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 30: 1 · Oct 31: 1 · Nov 1: 1 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xnz4jqcsqtwx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x18959b686ec0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xnz4jqcsqtwx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j2386432-mgmt-2/ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xnz4jqcsqtwx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j2386432-mgmt-2/ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x18959b686e80>{
observation: <*errors.errorString | 0x18959b944420>{
s: "dev/centralus/ci01-j2386432-mgmt-2/ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x18959b686de0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j2386432-mgmt-2\"; let ns = \"ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster\"; let start = datetime(2026-11-04T16:55:25.5270861Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x18959b162140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x18959b686da0>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j2386432-mgmt-2\"; let ns = \"ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster\"; let start = datetime(2026-11-04T16:55:25.5270861Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x18959d646120>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xnz4jqcsqtwx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x18959b686ec0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xnz4jqcsqtwx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j2386432-mgmt-2/ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xnz4jqcsqtwx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j2386432-mgmt-2/ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x18959b686e80>{
observation: <*errors.errorString | 0x18959b944420>{
s: "dev/centralus/ci01-j2386432-mgmt-2/ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x18959b686de0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j2386432-mgmt-2\"; let ns = \"ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster\"; let start = datetime(2026-11-04T16:55:25.5270861Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x18959b162140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x18959b686da0>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j2386432-mgmt-2\"; let ns = \"ocm-arohcpci01-2tshbcgnhburhvsb38erihusrs78ncpe-cluster\"; let start = datetime(2026-11-04T16:55:25.5270861Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x18959d646120>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4pvpz8jmv854/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x17d7881c8fc0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4pvpz8jmv854/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1986304-mgmt-2/ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4pvpz8jmv854/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1986304-mgmt-2/ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x17d7881c8fa0>{
observation: <*errors.errorString | 0x17d78767c210>{
s: "dev/centralus/ci01-j1986304-mgmt-2/ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x17d7881c8f80>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1986304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster\"; let start = datetime(2026-11-01T08:38:22.7146218Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x17d78849c410>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x17d7881c8f60>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1986304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster\"; let start = datetime(2026-11-01T08:38:22.7146218Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x17d789816420>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4pvpz8jmv854/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x17d7881c8fc0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4pvpz8jmv854/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1986304-mgmt-2/ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4pvpz8jmv854/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1986304-mgmt-2/ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x17d7881c8fa0>{
observation: <*errors.errorString | 0x17d78767c210>{
s: "dev/centralus/ci01-j1986304-mgmt-2/ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x17d7881c8f80>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1986304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster\"; let start = datetime(2026-11-01T08:38:22.7146218Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x17d78849c410>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x17d7881c8f60>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1986304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tqapdkr2ntohru8122gs6j85isvfana-cluster\"; let start = datetime(2026-11-01T08:38:22.7146218Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x17d789816420>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g4l4rfkwgs8r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x122e06162000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g4l4rfkwgs8r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j5335424-mgmt-1/ocm-arohcpci01-2tpgpa1lmdhqrcm5kuossbbertskr78b-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g4l4rfkwgs8r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j5335424-mgmt-1/ocm-arohcpci01-2tpgpa1lmdhqrcm5kuossbbertskr78b-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x122e07905760>{
s: "dev/centralus/ci01-j5335424-mgmt-1/ocm-arohcpci01-2tpgpa1lmdhqrcm5kuossbbertskr78b-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g4l4rfkwgs8r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x122e06162000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g4l4rfkwgs8r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j5335424-mgmt-1/ocm-arohcpci01-2tpgpa1lmdhqrcm5kuossbbertskr78b-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g4l4rfkwgs8r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j5335424-mgmt-1/ocm-arohcpci01-2tpgpa1lmdhqrcm5kuossbbertskr78b-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x122e07905760>{
s: "dev/centralus/ci01-j5335424-mgmt-1/ocm-arohcpci01-2tpgpa1lmdhqrcm5kuossbbertskr78b-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 1.14%3 of 264 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 2 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-grfgxfqsc7rp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x101b468e50e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-grfgxfqsc7rp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8117120-mgmt-2/ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-grfgxfqsc7rp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8117120-mgmt-2/ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x101b468e50a0>{
observation: <*errors.errorString | 0x101b484a1b20>{
s: "dev/centralus/ci01-j8117120-mgmt-2/ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x101b468e5080>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8117120-mgmt-2\"; let ns = \"ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster\"; let start = datetime(2026-11-03T13:19:21.3510455Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x101b464da4b0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x101b468e5000>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8117120-mgmt-2\"; let ns = \"ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster\"; let start = datetime(2026-11-03T13:19:21.3510455Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x101b47f637a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-grfgxfqsc7rp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x101b468e50e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-grfgxfqsc7rp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8117120-mgmt-2/ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-grfgxfqsc7rp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8117120-mgmt-2/ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x101b468e50a0>{
observation: <*errors.errorString | 0x101b484a1b20>{
s: "dev/centralus/ci01-j8117120-mgmt-2/ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x101b468e5080>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8117120-mgmt-2\"; let ns = \"ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster\"; let start = datetime(2026-11-03T13:19:21.3510455Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x101b464da4b0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x101b468e5000>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8117120-mgmt-2\"; let ns = \"ocm-arohcpci01-2trp3a60q9osug8b15jkt5c6ov2rjptf-cluster\"; let start = datetime(2026-11-03T13:19:21.3510455Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x101b47f637a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hbv8lppwpvdc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x61fcfc6720>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hbv8lppwpvdc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/westus3/ci01-j5200000-mgmt-1/ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hbv8lppwpvdc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/westus3/ci01-j5200000-mgmt-1/ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x61fcfc6700>{
observation: <*errors.errorString | 0x61fd1bb530>{
s: "dev/westus3/ci01-j5200000-mgmt-1/ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x61fcfc6660>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5200000-mgmt-1\"; let ns = \"ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster\"; let start = datetime(2026-11-03T11:28:43.6866081Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x61fd5ce140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x61fcfc6620>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5200000-mgmt-1\"; let ns = \"ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster\"; let start = datetime(2026-11-03T11:28:43.6866081Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x61fed100f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hbv8lppwpvdc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x61fcfc6720>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hbv8lppwpvdc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/westus3/ci01-j5200000-mgmt-1/ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hbv8lppwpvdc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/westus3/ci01-j5200000-mgmt-1/ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x61fcfc6700>{
observation: <*errors.errorString | 0x61fd1bb530>{
s: "dev/westus3/ci01-j5200000-mgmt-1/ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x61fcfc6660>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5200000-mgmt-1\"; let ns = \"ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster\"; let start = datetime(2026-11-03T11:28:43.6866081Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x61fd5ce140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x61fcfc6620>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5200000-mgmt-1\"; let ns = \"ocm-arohcpci01-2trnfaqrl9pbr0m6jhmpilc8d16uni12-cluster\"; let start = datetime(2026-11-03T11:28:43.6866081Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x61fed100f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-nkg2vsfh4k49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3b9f5c8c91a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-nkg2vsfh4k49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8819200-mgmt-2/ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-nkg2vsfh4k49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8819200-mgmt-2/ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x3b9f5c8c9180>{
observation: <*errors.errorString | 0x3b9f5c1d9220>{
s: "dev/centralus/ci01-j8819200-mgmt-2/ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x3b9f5c8c9160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8819200-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster\"; let start = datetime(2026-11-02T11:22:54.2015101Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3b9f5c41f180>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3b9f5c8c9140>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8819200-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster\"; let start = datetime(2026-11-02T11:22:54.2015101Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3b9f5db4e690>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-nkg2vsfh4k49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3b9f5c8c91a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-nkg2vsfh4k49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8819200-mgmt-2/ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-nkg2vsfh4k49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: dev/centralus/ci01-j8819200-mgmt-2/ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x3b9f5c8c9180>{
observation: <*errors.errorString | 0x3b9f5c1d9220>{
s: "dev/centralus/ci01-j8819200-mgmt-2/ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x3b9f5c8c9160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8819200-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster\"; let start = datetime(2026-11-02T11:22:54.2015101Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3b9f5c41f180>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3b9f5c8c9140>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8819200-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr29fh19lapohb9h48uiml4fi92ckfm-cluster\"; let start = datetime(2026-11-02T11:22:54.2015101Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3b9f5db4e690>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 3 | 1.14%3 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 2 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-1-5bpgxl"
Unexpected error:
<*errors.joinError | 0x1b487940348>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-5bpgxl-k6gdh-5h9dt egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-5bpgxl-k6gdh-5h9dt": nodes "cluster-np-5bpgxl-k6gdh-5h9dt" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1b48756bca0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-5bpgxl-k6gdh-5h9dt egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-5bpgxl-k6gdh-5h9dt\": nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1b48756bc80>{
msg: "failed to label node cluster-np-5bpgxl-k6gdh-5h9dt egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-5bpgxl-k6gdh-5h9dt\": nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1b48756bc60>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-5bpgxl-k6gdh-5h9dt\": nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1b48651fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-5bpgxl-k6gdh-5h9dt",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-1-5bpgxl"
Unexpected error:
<*errors.joinError | 0x1b487940348>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-5bpgxl-k6gdh-5h9dt egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-5bpgxl-k6gdh-5h9dt": nodes "cluster-np-5bpgxl-k6gdh-5h9dt" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1b48756bca0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-5bpgxl-k6gdh-5h9dt egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-5bpgxl-k6gdh-5h9dt\": nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1b48756bc80>{
msg: "failed to label node cluster-np-5bpgxl-k6gdh-5h9dt egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-5bpgxl-k6gdh-5h9dt\": nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1b48756bc60>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-5bpgxl-k6gdh-5h9dt\": nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1b48651fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-5bpgxl-k6gdh-5h9dt\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-5bpgxl-k6gdh-5h9dt",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-4-22-dw5jbr"
Unexpected error:
<*errors.joinError | 0x310344764678>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-dw5jbr-8xq5j-7njgd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-dw5jbr-8xq5j-7njgd": nodes "cluster-np-dw5jbr-8xq5j-7njgd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x310344418240>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-dw5jbr-8xq5j-7njgd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-dw5jbr-8xq5j-7njgd\": nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x310344418220>{
msg: "failed to label node cluster-np-dw5jbr-8xq5j-7njgd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-dw5jbr-8xq5j-7njgd\": nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x310344418200>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-dw5jbr-8xq5j-7njgd\": nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x3103442daa00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-dw5jbr-8xq5j-7njgd",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-4-22-dw5jbr"
Unexpected error:
<*errors.joinError | 0x310344764678>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-dw5jbr-8xq5j-7njgd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-dw5jbr-8xq5j-7njgd": nodes "cluster-np-dw5jbr-8xq5j-7njgd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x310344418240>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-dw5jbr-8xq5j-7njgd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-dw5jbr-8xq5j-7njgd\": nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x310344418220>{
msg: "failed to label node cluster-np-dw5jbr-8xq5j-7njgd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-dw5jbr-8xq5j-7njgd\": nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x310344418200>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-dw5jbr-8xq5j-7njgd\": nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x3103442daa00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-dw5jbr-8xq5j-7njgd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-dw5jbr-8xq5j-7njgd",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-1-smg52n"
Unexpected error:
<*errors.joinError | 0xbb1c755a3f0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-smg52n-rpl2b-5q7gb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-smg52n-rpl2b-5q7gb": nodes "cluster-np-smg52n-rpl2b-5q7gb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xbb1c6d07bc0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-smg52n-rpl2b-5q7gb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-smg52n-rpl2b-5q7gb\": nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xbb1c6d07b60>{
msg: "failed to label node cluster-np-smg52n-rpl2b-5q7gb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-smg52n-rpl2b-5q7gb\": nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xbb1c6d07ae0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-smg52n-rpl2b-5q7gb\": nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xbb1c76a45a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-smg52n-rpl2b-5q7gb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-1-smg52n"
Unexpected error:
<*errors.joinError | 0xbb1c755a3f0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-smg52n-rpl2b-5q7gb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-smg52n-rpl2b-5q7gb": nodes "cluster-np-smg52n-rpl2b-5q7gb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xbb1c6d07bc0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-smg52n-rpl2b-5q7gb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-smg52n-rpl2b-5q7gb\": nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xbb1c6d07b60>{
msg: "failed to label node cluster-np-smg52n-rpl2b-5q7gb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-smg52n-rpl2b-5q7gb\": nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xbb1c6d07ae0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-smg52n-rpl2b-5q7gb\": nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xbb1c76a45a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-smg52n-rpl2b-5q7gb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-smg52n-rpl2b-5q7gb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
alert [svc] ClusterCredentialMetricMissing firedfull failure pattern: alert [svc] ClusterCredentialMetricMissing fired Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | alert | 3 | 1.14%3 of 264 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 2 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-02T05:22:21Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j7644032-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j7644032-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-02T00:53:49Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j4763648-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j4763648-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-01T21:24:27Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j4962816-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j4962816-svc. Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
E2E Jobs
39
Success Rate
74.36%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.96.67% (29/30)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 80.00% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster then delete it by deleting the customer resource group | |
| 84.44% | 45 | integration/parallel | Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully | |
| 84.44% | 45 | integration/parallel | Customer should be able to update node pool labels and taints |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 4 | 10.26%4 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 18 · Nov 4: 6 · Nov 5: 0 · Nov 6: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x302ab7a3aed0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x302ab7a3aed0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.004s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1ad52dbe78f0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.004s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1ad52dbe78f0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2d462105c590>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2d462105c590>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]full failure pattern: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | unknown | 4 | 10.26%4 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 3 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 635.698s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]
Expected
<[]string | len:1, cap:1>: [
"image-registry (Available=False)",
]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 635.698s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]
Expected
<[]string | len:1, cap:1>: [
"image-registry (Available=False)",
]
to be emptyfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 638.203s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]
Expected
<[]string | len:1, cap:1>: [
"image-registry (Available=False)",
]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 638.203s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]
Expected
<[]string | len:1, cap:1>: [
"image-registry (Available=False)",
]
to be emptyfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 635.300s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]
Expected
<[]string | len:1, cap:1>: [
"image-registry (Available=False)",
]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 635.300s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)]
Expected
<[]string | len:1, cap:1>: [
"image-registry (Available=False)",
]
to be emptyContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for nodepool="<nodepool>" in cluster="<cluster>" resourcegroup="<resource-group>" to finish deleting, caused by: timeout '25full failure pattern: failed waiting for nodepool="<nodepool>" in cluster="<cluster>" resourcegroup="<resource-group>" to finish deleting, caused by: timeout '25 Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | unknown | 3 | 7.69%3 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 2 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:160]: failed to delete GPU nodepool gpu-np-1
Expected success, but got an error:
<*fmt.wrapErrors | 0x2cd746b37560>:
failed waiting for nodepool="gpu-np-1" in cluster="cluster-gpu-np" resourcegroup="rg-gpu-nodepool-pr8xrqb4zgmp" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-pr8xrqb4zgmp, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"gpu-np-1\" in cluster=\"cluster-gpu-np\" resourcegroup=\"rg-gpu-nodepool-pr8xrqb4zgmp\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-pr8xrqb4zgmp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2cd746b8d8f0>{
s: "timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-pr8xrqb4zgmp",
},
<context.deadlineExceededError>{},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:160]: failed to delete GPU nodepool gpu-np-1
Expected success, but got an error:
<*fmt.wrapErrors | 0x2cd746b37560>:
failed waiting for nodepool="gpu-np-1" in cluster="cluster-gpu-np" resourcegroup="rg-gpu-nodepool-pr8xrqb4zgmp" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-pr8xrqb4zgmp, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"gpu-np-1\" in cluster=\"cluster-gpu-np\" resourcegroup=\"rg-gpu-nodepool-pr8xrqb4zgmp\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-pr8xrqb4zgmp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2cd746b8d8f0>{
s: "timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-pr8xrqb4zgmp",
},
<context.deadlineExceededError>{},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:160]: failed to delete GPU nodepool gpu-np-1
Expected success, but got an error:
<*fmt.wrapErrors | 0xe3a0690660>:
failed waiting for nodepool="gpu-np-1" in cluster="cluster-gpu-np" resourcegroup="rg-gpu-nodepool-ljfz7k4xghhj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-ljfz7k4xghhj, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"gpu-np-1\" in cluster=\"cluster-gpu-np\" resourcegroup=\"rg-gpu-nodepool-ljfz7k4xghhj\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-ljfz7k4xghhj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xe39e7b4a60>{
s: "timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-ljfz7k4xghhj",
},
<context.deadlineExceededError>{},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:160]: failed to delete GPU nodepool gpu-np-1
Expected success, but got an error:
<*fmt.wrapErrors | 0xe3a0690660>:
failed waiting for nodepool="gpu-np-1" in cluster="cluster-gpu-np" resourcegroup="rg-gpu-nodepool-ljfz7k4xghhj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-ljfz7k4xghhj, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"gpu-np-1\" in cluster=\"cluster-gpu-np\" resourcegroup=\"rg-gpu-nodepool-ljfz7k4xghhj\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-ljfz7k4xghhj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xe39e7b4a60>{
s: "timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-ljfz7k4xghhj",
},
<context.deadlineExceededError>{},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:160]: failed to delete GPU nodepool gpu-np-1
Expected success, but got an error:
<*fmt.wrapErrors | 0x1c7f66094fc0>:
failed waiting for nodepool="gpu-np-1" in cluster="cluster-gpu-np" resourcegroup="rg-gpu-nodepool-bm24jd94dtj2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-bm24jd94dtj2, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"gpu-np-1\" in cluster=\"cluster-gpu-np\" resourcegroup=\"rg-gpu-nodepool-bm24jd94dtj2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-bm24jd94dtj2, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1c7f674ee580>{
s: "timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-bm24jd94dtj2",
},
<context.deadlineExceededError>{},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:160]: failed to delete GPU nodepool gpu-np-1
Expected success, but got an error:
<*fmt.wrapErrors | 0x1c7f66094fc0>:
failed waiting for nodepool="gpu-np-1" in cluster="cluster-gpu-np" resourcegroup="rg-gpu-nodepool-bm24jd94dtj2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-bm24jd94dtj2, error: context deadline exceeded
{
msg: "failed waiting for nodepool=\"gpu-np-1\" in cluster=\"cluster-gpu-np\" resourcegroup=\"rg-gpu-nodepool-bm24jd94dtj2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-bm24jd94dtj2, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1c7f674ee580>{
s: "timeout '25.000000' minutes exceeded during DeleteNodePool for nodepool gpu-np-1 in cluster cluster-gpu-np in resource group rg-gpu-nodepool-bm24jd94dtj2",
},
<context.deadlineExceededError>{},
],
}Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||
Deserialization Error: no output from commandfull failure pattern: Deserialization Error: no output from command Signal: NoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior history | unknown | 2 | 5.13%2 of 39 job runs affected | NoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 2 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1194 +0x745 fp=0x91252cc7f00 sp=0x91252cc7b88 pc=0xc3ee7c5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).collectHCPInspectData.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1128 +0x66 fp=0x91252cc7f78 sp=0x91252cc7f00 pc=0xc3edfe6
golang.org/x/sync/errgroup.(*Group).Go.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50 fp=0x91252cc7fe0 sp=0x91252cc7f78 pc=0x6bcc930
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x91252cc7fe8 sp=0x91252cc7fe0 pc=0x5672501
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2962
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
goroutine 2964 gp=0x91252ca0000 m=nil [IO wait]:
runtime.gopark(0x91251cc7000?, 0x912534a7be0?, 0xcc?, 0x6f?, 0xb?)
/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x912534a7ba8 sp=0x912534a7b88 pc=0x566a28e
runtime.netpollblock(0x5692958?, 0x55f0426?, 0x0?)
/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0x912534a7be0 sp=0x912534a7ba8 pc=0x5628f97
internal/poll.runtime_pollWait(0x7fa7a79c6800, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0x912534a7c00 sp=0x912534a7be0 pc=0x5668b25
internal/poll.(*pollDesc).wait(0x91252962a20?, 0x91251cc7364?, 0x1)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0x912534a7c28 sp=0x912534a7c00 pc=0x56f0f27
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x91252962a20, {0x91251cc7364, 0x49c, 0x49c})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x2ae fp=0x912534a7cc0 sp=0x912534a7c28 pc=0x56f228e
os.(*File).read(...)
/usr/lib/golang/src/os/file_posix.go:30
os.(*File).Read(0x9125580a7d0, {0x91251cc7364?, 0x55ec7ff?, 0x209?})
/usr/lib/golang/src/os/file.go:144 +0x4f fp=0x912534a7d00 sp=0x912534a7cc0 pc=0x56fe6ef
os.fileWithoutWriteTo.Read({{}, 0x91255155680?}, {0x91251cc7364?, 0x55f9cdf?, 0xdda41e0?})
<autogenerated>:1 +0x18 fp=0x912534a7d30 sp=0x912534a7d00 pc=0x57139f8
bytes.(*Buffer).ReadFrom(0x91255155680, {0x104e8020, 0x9125580a7d0})
/usr/lib/golang/src/bytes/buffer.go:229 +0x98 fp=0x912534a7d88 sp=0x912534a7d30 pc=0x5732ab8
io.copyBuffer({0x104ec140, 0x91255155680}, {0x104e8020, 0x9125580a7d0}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:415 +0x151 fp=0x912534a7e00 sp=0x912534a7d88 pc=0x56e47f1
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os.genericWriteTo(0x9125580a7d0?, {0x104ec140?, 0x91255155680?})
/usr/lib/golang/src/os/file.go:295 +0x35 fp=0x912534a7e48 sp=0x912534a7e00 pc=0x56ff015
os.(*File).WriteTo(0x9125580a7d0, {0x104ec140, 0x91255155680})
/usr/lib/golang/src/os/file.go:273 +0x9c fp=0x912534a7e88 sp=0x912534a7e48 pc=0x56fef5c
io.copyBuffer({0x104ec140, 0x91255155680}, {0x104e7ea0, 0x9125580a7d0}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:411 +0x9d fp=0x912534a7f00 sp=0x912534a7e88 pc=0x56e473d
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os/exec.(*Cmd).writerDescriptor.func1()
/usr/lib/golang/src/os/exec/exec.go:602 +0x34 fp=0x912534a7f60 sp=0x912534a7f00 pc=0x5e58e74
os/exec.(*Cmd).Start.func2(0x106d33d0?)
/usr/lib/golang/src/os/exec/exec.go:757 +0x2c fp=0x912534a7fc8 sp=0x912534a7f60 pc=0x5e59a4c
os/exec.(*Cmd).Start.gowrap1()
/usr/lib/golang/src/os/exec/exec.go:769 +0x1b fp=0x912534a7fe0 sp=0x912534a7fc8 pc=0x5e599fb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x912534a7fe8 sp=0x912534a7fe0 pc=0x5672501
created by os/exec.(*Cmd).Start in goroutine 2963
/usr/lib/golang/src/os/exec/exec.go:756 +0x92e
goroutine 2965 gp=0x91252ca01e0 m=nil [select, 2 minutes]:
runtime.gopark(0x91251b97738?, 0x2?, 0x1?, 0x76?, 0x91251b976bc?)
/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x91251b97550 sp=0x91251b97530 pc=0x566a28e
runtime.selectgo(0x91251b97738, 0x91251b976b8, 0x91251b976d8?, 0x1, 0x105f84d8?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0xaa5 fp=0x91251b97680 sp=0x91251b97550 pc=0x5644965
os/exec.(*Cmd).watchCtx(0x91252a521a0, 0x912526d57a0)
/usr/lib/golang/src/os/exec/exec.go:797 +0xb1 fp=0x91251b977c0 sp=0x91251b97680 pc=0x5e59d11
os/exec.(*Cmd).Start.gowrap2()
/usr/lib/golang/src/os/exec/exec.go:783 +0x1b fp=0x91251b977e0 sp=0x91251b977c0 pc=0x5e599bb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x91251b977e8 sp=0x91251b977e0 pc=0x5672501
created by os/exec.(*Cmd).Start in goroutine 2963
/usr/lib/golang/src/os/exec/exec.go:783 +0x8f5
rax 0xca
rbx 0x0
rcx 0x5674043
rdx 0x0
rdi 0x17607cf8
rsi 0x80
rbp 0x7ffcae4f7150
rsp 0x7ffcae4f7108
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x91251b42008
r13 0x1
r14 0x17605860
r15 0x3
rip 0x5674041
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from command
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1194 +0x745 fp=0x91252cc7f00 sp=0x91252cc7b88 pc=0xc3ee7c5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).collectHCPInspectData.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1128 +0x66 fp=0x91252cc7f78 sp=0x91252cc7f00 pc=0xc3edfe6
golang.org/x/sync/errgroup.(*Group).Go.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50 fp=0x91252cc7fe0 sp=0x91252cc7f78 pc=0x6bcc930
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x91252cc7fe8 sp=0x91252cc7fe0 pc=0x5672501
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2962
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
goroutine 2964 gp=0x91252ca0000 m=nil [IO wait]:
runtime.gopark(0x91251cc7000?, 0x912534a7be0?, 0xcc?, 0x6f?, 0xb?)
/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x912534a7ba8 sp=0x912534a7b88 pc=0x566a28e
runtime.netpollblock(0x5692958?, 0x55f0426?, 0x0?)
/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0x912534a7be0 sp=0x912534a7ba8 pc=0x5628f97
internal/poll.runtime_pollWait(0x7fa7a79c6800, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0x912534a7c00 sp=0x912534a7be0 pc=0x5668b25
internal/poll.(*pollDesc).wait(0x91252962a20?, 0x91251cc7364?, 0x1)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0x912534a7c28 sp=0x912534a7c00 pc=0x56f0f27
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x91252962a20, {0x91251cc7364, 0x49c, 0x49c})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x2ae fp=0x912534a7cc0 sp=0x912534a7c28 pc=0x56f228e
os.(*File).read(...)
/usr/lib/golang/src/os/file_posix.go:30
os.(*File).Read(0x9125580a7d0, {0x91251cc7364?, 0x55ec7ff?, 0x209?})
/usr/lib/golang/src/os/file.go:144 +0x4f fp=0x912534a7d00 sp=0x912534a7cc0 pc=0x56fe6ef
os.fileWithoutWriteTo.Read({{}, 0x91255155680?}, {0x91251cc7364?, 0x55f9cdf?, 0xdda41e0?})
<autogenerated>:1 +0x18 fp=0x912534a7d30 sp=0x912534a7d00 pc=0x57139f8
bytes.(*Buffer).ReadFrom(0x91255155680, {0x104e8020, 0x9125580a7d0})
/usr/lib/golang/src/bytes/buffer.go:229 +0x98 fp=0x912534a7d88 sp=0x912534a7d30 pc=0x5732ab8
io.copyBuffer({0x104ec140, 0x91255155680}, {0x104e8020, 0x9125580a7d0}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:415 +0x151 fp=0x912534a7e00 sp=0x912534a7d88 pc=0x56e47f1
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os.genericWriteTo(0x9125580a7d0?, {0x104ec140?, 0x91255155680?})
/usr/lib/golang/src/os/file.go:295 +0x35 fp=0x912534a7e48 sp=0x912534a7e00 pc=0x56ff015
os.(*File).WriteTo(0x9125580a7d0, {0x104ec140, 0x91255155680})
/usr/lib/golang/src/os/file.go:273 +0x9c fp=0x912534a7e88 sp=0x912534a7e48 pc=0x56fef5c
io.copyBuffer({0x104ec140, 0x91255155680}, {0x104e7ea0, 0x9125580a7d0}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:411 +0x9d fp=0x912534a7f00 sp=0x912534a7e88 pc=0x56e473d
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os/exec.(*Cmd).writerDescriptor.func1()
/usr/lib/golang/src/os/exec/exec.go:602 +0x34 fp=0x912534a7f60 sp=0x912534a7f00 pc=0x5e58e74
os/exec.(*Cmd).Start.func2(0x106d33d0?)
/usr/lib/golang/src/os/exec/exec.go:757 +0x2c fp=0x912534a7fc8 sp=0x912534a7f60 pc=0x5e59a4c
os/exec.(*Cmd).Start.gowrap1()
/usr/lib/golang/src/os/exec/exec.go:769 +0x1b fp=0x912534a7fe0 sp=0x912534a7fc8 pc=0x5e599fb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x912534a7fe8 sp=0x912534a7fe0 pc=0x5672501
created by os/exec.(*Cmd).Start in goroutine 2963
/usr/lib/golang/src/os/exec/exec.go:756 +0x92e
goroutine 2965 gp=0x91252ca01e0 m=nil [select, 2 minutes]:
runtime.gopark(0x91251b97738?, 0x2?, 0x1?, 0x76?, 0x91251b976bc?)
/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x91251b97550 sp=0x91251b97530 pc=0x566a28e
runtime.selectgo(0x91251b97738, 0x91251b976b8, 0x91251b976d8?, 0x1, 0x105f84d8?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0xaa5 fp=0x91251b97680 sp=0x91251b97550 pc=0x5644965
os/exec.(*Cmd).watchCtx(0x91252a521a0, 0x912526d57a0)
/usr/lib/golang/src/os/exec/exec.go:797 +0xb1 fp=0x91251b977c0 sp=0x91251b97680 pc=0x5e59d11
os/exec.(*Cmd).Start.gowrap2()
/usr/lib/golang/src/os/exec/exec.go:783 +0x1b fp=0x91251b977e0 sp=0x91251b977c0 pc=0x5e599bb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x91251b977e8 sp=0x91251b977e0 pc=0x5672501
created by os/exec.(*Cmd).Start in goroutine 2963
/usr/lib/golang/src/os/exec/exec.go:783 +0x8f5
rax 0xca
rbx 0x0
rcx 0x5674043
rdx 0x0
rdi 0x17607cf8
rsi 0x80
rbp 0x7ffcae4f7150
rsp 0x7ffcae4f7108
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x91251b42008
r13 0x1
r14 0x17605860
r15 0x3
rip 0x5674041
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from commandlib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0x13bfb01a5be0 sp=0x13bfb01a5ba8 pc=0x5628f97
internal/poll.runtime_pollWait(0x7f298082dc00, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0x13bfb01a5c00 sp=0x13bfb01a5be0 pc=0x5668b25
internal/poll.(*pollDesc).wait(0x13bfb292fbc0?, 0x13bfb0e7ab94?, 0x1)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0x13bfb01a5c28 sp=0x13bfb01a5c00 pc=0x56f0f27
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x13bfb292fbc0, {0x13bfb0e7ab94, 0x46c, 0x46c})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x2ae fp=0x13bfb01a5cc0 sp=0x13bfb01a5c28 pc=0x56f228e
os.(*File).read(...)
/usr/lib/golang/src/os/file_posix.go:30
os.(*File).Read(0x13bfb2050798, {0x13bfb0e7ab94?, 0x58a2e00?, 0x209?})
/usr/lib/golang/src/os/file.go:144 +0x4f fp=0x13bfb01a5d00 sp=0x13bfb01a5cc0 pc=0x56fe6ef
os.fileWithoutWriteTo.Read({{}, 0x13bfb0d0c6f0?}, {0x13bfb0e7ab94?, 0x55f9cdf?, 0xdda41e0?})
<autogenerated>:1 +0x18 fp=0x13bfb01a5d30 sp=0x13bfb01a5d00 pc=0x57139f8
bytes.(*Buffer).ReadFrom(0x13bfb0d0c6f0, {0x104e8020, 0x13bfb2050798})
/usr/lib/golang/src/bytes/buffer.go:229 +0x98 fp=0x13bfb01a5d88 sp=0x13bfb01a5d30 pc=0x5732ab8
io.copyBuffer({0x104ec140, 0x13bfb0d0c6f0}, {0x104e8020, 0x13bfb2050798}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:415 +0x151 fp=0x13bfb01a5e00 sp=0x13bfb01a5d88 pc=0x56e47f1
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os.genericWriteTo(0x13bfb2050798?, {0x104ec140?, 0x13bfb0d0c6f0?})
/usr/lib/golang/src/os/file.go:295 +0x35 fp=0x13bfb01a5e48 sp=0x13bfb01a5e00 pc=0x56ff015
os.(*File).WriteTo(0x13bfb2050798, {0x104ec140, 0x13bfb0d0c6f0})
/usr/lib/golang/src/os/file.go:273 +0x9c fp=0x13bfb01a5e88 sp=0x13bfb01a5e48 pc=0x56fef5c
io.copyBuffer({0x104ec140, 0x13bfb0d0c6f0}, {0x104e7ea0, 0x13bfb2050798}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:411 +0x9d fp=0x13bfb01a5f00 sp=0x13bfb01a5e88 pc=0x56e473d
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os/exec.(*Cmd).writerDescriptor.func1()
/usr/lib/golang/src/os/exec/exec.go:602 +0x34 fp=0x13bfb01a5f60 sp=0x13bfb01a5f00 pc=0x5e58e74
os/exec.(*Cmd).Start.func2(0x13bfb01a5fa8?)
/usr/lib/golang/src/os/exec/exec.go:757 +0x2c fp=0x13bfb01a5fc8 sp=0x13bfb01a5f60 pc=0x5e59a4c
os/exec.(*Cmd).Start.gowrap1()
/usr/lib/golang/src/os/exec/exec.go:769 +0x1b fp=0x13bfb01a5fe0 sp=0x13bfb01a5fc8 pc=0x5e599fb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x13bfb01a5fe8 sp=0x13bfb01a5fe0 pc=0x5672501
created by os/exec.(*Cmd).Start in goroutine 2900
/usr/lib/golang/src/os/exec/exec.go:756 +0x92e
goroutine 2899 gp=0x13bfb01db4a0 m=nil [sync.WaitGroup.Wait, 2 minutes]:
runtime.gopark(0x13bfb07cf018?, 0x55fce54?, 0x40?, 0x53?, 0x7f29783781f8?)
/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x13bfb07cefb8 sp=0x13bfb07cef98 pc=0x566a28e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x13bfb11e5310, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x13bfb07cf020 sp=0x13bfb07cefb8 pc=0x5645972
sync.runtime_SemacquireWaitGroup(0x13bfaf9fbe90?, 0xa0?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0x13bfb07cf058 sp=0x13bfb07cf020 pc=0x566bc8e
sync.(*WaitGroup).Wait(0x13bfb11e5308)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0x13bfb07cf080 sp=0x13bfb07cf058 pc=0x56816a5
golang.org/x/sync/errgroup.(*Group).Wait(0x13bfb11e5300)
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x1e fp=0x13bfb07cf0a0 sp=0x13bfb07cf080 pc=0x6bcc7de
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).collectHCPInspectData(0x13bfb0046b40, {0x105cf2b0, 0x13bfb02bb1d0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1132 +0x4eb fp=0x13bfb07cff18 sp=0x13bfb07cf0a0 pc=0xc3edf4b
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).collectDebugInfo.func3()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:536 +0x65 fp=0x13bfb07cff78 sp=0x13bfb07cff18 pc=0xc3e6805
golang.org/x/sync/errgroup.(*Group).Go.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50 fp=0x13bfb07cffe0 sp=0x13bfb07cff78 pc=0x6bcc930
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x13bfb07cffe8 sp=0x13bfb07cffe0 pc=0x5672501
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2880
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
rax 0xca
rbx 0x0
rcx 0x5674043
rdx 0x0
rdi 0x17607cf8
rsi 0x80
rbp 0x7ffd947d7538
rsp 0x7ffd947d74f0
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x13bfaf940008
r13 0x1
r14 0x17605860
r15 0x1
rip 0x5674041
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from command
lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0x13bfb01a5be0 sp=0x13bfb01a5ba8 pc=0x5628f97
internal/poll.runtime_pollWait(0x7f298082dc00, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0x13bfb01a5c00 sp=0x13bfb01a5be0 pc=0x5668b25
internal/poll.(*pollDesc).wait(0x13bfb292fbc0?, 0x13bfb0e7ab94?, 0x1)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0x13bfb01a5c28 sp=0x13bfb01a5c00 pc=0x56f0f27
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x13bfb292fbc0, {0x13bfb0e7ab94, 0x46c, 0x46c})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x2ae fp=0x13bfb01a5cc0 sp=0x13bfb01a5c28 pc=0x56f228e
os.(*File).read(...)
/usr/lib/golang/src/os/file_posix.go:30
os.(*File).Read(0x13bfb2050798, {0x13bfb0e7ab94?, 0x58a2e00?, 0x209?})
/usr/lib/golang/src/os/file.go:144 +0x4f fp=0x13bfb01a5d00 sp=0x13bfb01a5cc0 pc=0x56fe6ef
os.fileWithoutWriteTo.Read({{}, 0x13bfb0d0c6f0?}, {0x13bfb0e7ab94?, 0x55f9cdf?, 0xdda41e0?})
<autogenerated>:1 +0x18 fp=0x13bfb01a5d30 sp=0x13bfb01a5d00 pc=0x57139f8
bytes.(*Buffer).ReadFrom(0x13bfb0d0c6f0, {0x104e8020, 0x13bfb2050798})
/usr/lib/golang/src/bytes/buffer.go:229 +0x98 fp=0x13bfb01a5d88 sp=0x13bfb01a5d30 pc=0x5732ab8
io.copyBuffer({0x104ec140, 0x13bfb0d0c6f0}, {0x104e8020, 0x13bfb2050798}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:415 +0x151 fp=0x13bfb01a5e00 sp=0x13bfb01a5d88 pc=0x56e47f1
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os.genericWriteTo(0x13bfb2050798?, {0x104ec140?, 0x13bfb0d0c6f0?})
/usr/lib/golang/src/os/file.go:295 +0x35 fp=0x13bfb01a5e48 sp=0x13bfb01a5e00 pc=0x56ff015
os.(*File).WriteTo(0x13bfb2050798, {0x104ec140, 0x13bfb0d0c6f0})
/usr/lib/golang/src/os/file.go:273 +0x9c fp=0x13bfb01a5e88 sp=0x13bfb01a5e48 pc=0x56fef5c
io.copyBuffer({0x104ec140, 0x13bfb0d0c6f0}, {0x104e7ea0, 0x13bfb2050798}, {0x0, 0x0, 0x0})
/usr/lib/golang/src/io/io.go:411 +0x9d fp=0x13bfb01a5f00 sp=0x13bfb01a5e88 pc=0x56e473d
io.Copy(...)
/usr/lib/golang/src/io/io.go:388
os/exec.(*Cmd).writerDescriptor.func1()
/usr/lib/golang/src/os/exec/exec.go:602 +0x34 fp=0x13bfb01a5f60 sp=0x13bfb01a5f00 pc=0x5e58e74
os/exec.(*Cmd).Start.func2(0x13bfb01a5fa8?)
/usr/lib/golang/src/os/exec/exec.go:757 +0x2c fp=0x13bfb01a5fc8 sp=0x13bfb01a5f60 pc=0x5e59a4c
os/exec.(*Cmd).Start.gowrap1()
/usr/lib/golang/src/os/exec/exec.go:769 +0x1b fp=0x13bfb01a5fe0 sp=0x13bfb01a5fc8 pc=0x5e599fb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x13bfb01a5fe8 sp=0x13bfb01a5fe0 pc=0x5672501
created by os/exec.(*Cmd).Start in goroutine 2900
/usr/lib/golang/src/os/exec/exec.go:756 +0x92e
goroutine 2899 gp=0x13bfb01db4a0 m=nil [sync.WaitGroup.Wait, 2 minutes]:
runtime.gopark(0x13bfb07cf018?, 0x55fce54?, 0x40?, 0x53?, 0x7f29783781f8?)
/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x13bfb07cefb8 sp=0x13bfb07cef98 pc=0x566a28e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x13bfb11e5310, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x13bfb07cf020 sp=0x13bfb07cefb8 pc=0x5645972
sync.runtime_SemacquireWaitGroup(0x13bfaf9fbe90?, 0xa0?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0x13bfb07cf058 sp=0x13bfb07cf020 pc=0x566bc8e
sync.(*WaitGroup).Wait(0x13bfb11e5308)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0x13bfb07cf080 sp=0x13bfb07cf058 pc=0x56816a5
golang.org/x/sync/errgroup.(*Group).Wait(0x13bfb11e5300)
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x1e fp=0x13bfb07cf0a0 sp=0x13bfb07cf080 pc=0x6bcc7de
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).collectHCPInspectData(0x13bfb0046b40, {0x105cf2b0, 0x13bfb02bb1d0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1132 +0x4eb fp=0x13bfb07cff18 sp=0x13bfb07cf0a0 pc=0xc3edf4b
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).collectDebugInfo.func3()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:536 +0x65 fp=0x13bfb07cff78 sp=0x13bfb07cff18 pc=0xc3e6805
golang.org/x/sync/errgroup.(*Group).Go.func1()
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50 fp=0x13bfb07cffe0 sp=0x13bfb07cff78 pc=0x6bcc930
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x13bfb07cffe8 sp=0x13bfb07cffe0 pc=0x5672501
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 2880
/opt/app-root/src/github.com/Azure/ARO-HCP/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
rax 0xca
rbx 0x0
rcx 0x5674043
rdx 0x0
rdi 0x17607cf8
rsi 0x80
rbp 0x7ffd947d7538
rsp 0x7ffd947d74f0
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x13bfaf940008
r13 0x1
r14 0x17605860
r15 0x1
rip 0x5674041
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from commandContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode, delay likely due to PodDisruptionBudgets; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | unknown | 2 | 5.13%2 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:181]: failed to scale down node pool np-update-NODES from 3 to 2 replicas
Unexpected error:
<*fmt.wrapError | 0x3ecee624a820>:
failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-bg7nv5wx2hfw" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/331b3c36-db32-4412-8ce6-88847172068a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/331b3c36-db32-4412-8ce6-88847172068a",
"name": "331b3c36-db32-4412-8ce6-88847172068a",
"status": "Failed",
"startTime": "2026-11-04T02:23:14.394824785Z",
"endTime": "2026-11-04T02:42:23.578826333Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-np-update-nodes-qstbd-5tm8r: NotReady: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode, delay likely due to PodDisruptionBudgets\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/331b3c36-db32-4412-8ce6-88847172068a
correlation ID: a8466760-4916-480a-a847-4dbf9614228c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:181]: failed to scale down node pool np-update-NODES from 3 to 2 replicas
Unexpected error:
<*fmt.wrapError | 0x3ecee624a820>:
failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-bg7nv5wx2hfw" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/331b3c36-db32-4412-8ce6-88847172068a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/331b3c36-db32-4412-8ce6-88847172068a",
"name": "331b3c36-db32-4412-8ce6-88847172068a",
"status": "Failed",
"startTime": "2026-11-04T02:23:14.394824785Z",
"endTime": "2026-11-04T02:42:23.578826333Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-np-update-nodes-qstbd-5tm8r: NotReady: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode, delay likely due to PodDisruptionBudgets\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/331b3c36-db32-4412-8ce6-88847172068a
correlation ID: a8466760-4916-480a-a847-4dbf9614228c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:181]: failed to scale down node pool np-update-NODES from 3 to 2 replicas
Unexpected error:
<*fmt.wrapError | 0x3d43a5b88e40>:
failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-n67hl2lzfzqg" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20c426c1-fb38-4782-833d-56cd758439df
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20c426c1-fb38-4782-833d-56cd758439df",
"name": "20c426c1-fb38-4782-833d-56cd758439df",
"status": "Failed",
"startTime": "2026-11-04T00:43:17.789023656Z",
"endTime": "2026-11-04T01:02:21.220271762Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-np-update-nodes-sjc6j-llpnt: NotReady: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode, delay likely due to PodDisruptionBudgets\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20c426c1-fb38-4782-833d-56cd758439df
correlation ID: 5b036b02-212f-4bdc-bb8b-735dcfa65c2a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:181]: failed to scale down node pool np-update-NODES from 3 to 2 replicas
Unexpected error:
<*fmt.wrapError | 0x3d43a5b88e40>:
failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-n67hl2lzfzqg" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20c426c1-fb38-4782-833d-56cd758439df
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20c426c1-fb38-4782-833d-56cd758439df",
"name": "20c426c1-fb38-4782-833d-56cd758439df",
"status": "Failed",
"startTime": "2026-11-04T00:43:17.789023656Z",
"endTime": "2026-11-04T01:02:21.220271762Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-np-update-nodes-sjc6j-llpnt: NotReady: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode, delay likely due to PodDisruptionBudgets\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20c426c1-fb38-4782-833d-56cd758439df
correlation ID: 5b036b02-212f-4bdc-bb8b-735dcfa65c2a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 3 · Nov 5: 0 · Nov 6: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-pnnwrkwzdl95/cluster-candidate-4-22-8hfmn6 should provision
Unexpected error:
<*fmt.wrapError | 0x38e291eaa9a0>:
failed to create HCP cluster cluster-candidate-4-22-8hfmn6: failed waiting for cluster="cluster-candidate-4-22-8hfmn6" in resourcegroup="rg-candidate-4-22-pnnwrkwzdl95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2502c910-a037-445b-b907-e585d96bf1a2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2502c910-a037-445b-b907-e585d96bf1a2",
"name": "2502c910-a037-445b-b907-e585d96bf1a2",
"status": "Failed",
"startTime": "2026-11-04T15:35:00.919442516Z",
"endTime": "2026-11-04T16:04:08.909485949Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2502c910-a037-445b-b907-e585d96bf1a2
correlation ID: 66009ad0-92d4-401e-abaa-e67996fdf6ab
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-pnnwrkwzdl95/cluster-candidate-4-22-8hfmn6 should provision
Unexpected error:
<*fmt.wrapError | 0x38e291eaa9a0>:
failed to create HCP cluster cluster-candidate-4-22-8hfmn6: failed waiting for cluster="cluster-candidate-4-22-8hfmn6" in resourcegroup="rg-candidate-4-22-pnnwrkwzdl95" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2502c910-a037-445b-b907-e585d96bf1a2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2502c910-a037-445b-b907-e585d96bf1a2",
"name": "2502c910-a037-445b-b907-e585d96bf1a2",
"status": "Failed",
"startTime": "2026-11-04T15:35:00.919442516Z",
"endTime": "2026-11-04T16:04:08.909485949Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2502c910-a037-445b-b907-e585d96bf1a2
correlation ID: 66009ad0-92d4-401e-abaa-e67996fdf6ab
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x3501e9a2400>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-trs7hs9574fl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f614c834-9875-4f3e-894c-6f28c8c5194b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f614c834-9875-4f3e-894c-6f28c8c5194b",
"name": "f614c834-9875-4f3e-894c-6f28c8c5194b",
"status": "Failed",
"startTime": "2026-11-04T15:37:37.319368437Z",
"endTime": "2026-11-04T16:06:45.91009535Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f614c834-9875-4f3e-894c-6f28c8c5194b
correlation ID: d84990c2-82ec-4fbe-af84-ddff4074e113
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x3501e9a2400>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-trs7hs9574fl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f614c834-9875-4f3e-894c-6f28c8c5194b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f614c834-9875-4f3e-894c-6f28c8c5194b",
"name": "f614c834-9875-4f3e-894c-6f28c8c5194b",
"status": "Failed",
"startTime": "2026-11-04T15:37:37.319368437Z",
"endTime": "2026-11-04T16:06:45.91009535Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f614c834-9875-4f3e-894c-6f28c8c5194b
correlation ID: d84990c2-82ec-4fbe-af84-ddff4074e113
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-23-vl84tq" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x1983433aee20>:
failed to create HCP cluster cp-ystream-deny-pods-4-23-vl84tq: failed waiting for cluster="cp-ystream-deny-pods-4-23-vl84tq" in resourcegroup="rg-cp-ystream-deny-pods-4-23-7lr285dvx2pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8bd3c422-2dfd-4f3c-9840-80142bd1a9bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8bd3c422-2dfd-4f3c-9840-80142bd1a9bf",
"name": "8bd3c422-2dfd-4f3c-9840-80142bd1a9bf",
"status": "Failed",
"startTime": "2026-11-04T15:36:49.872179674Z",
"endTime": "2026-11-04T16:05:58.807973445Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8bd3c422-2dfd-4f3c-9840-80142bd1a9bf
correlation ID: 40a4cd7e-ed59-441d-8424-9d083fa24b5e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-23-vl84tq" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x1983433aee20>:
failed to create HCP cluster cp-ystream-deny-pods-4-23-vl84tq: failed waiting for cluster="cp-ystream-deny-pods-4-23-vl84tq" in resourcegroup="rg-cp-ystream-deny-pods-4-23-7lr285dvx2pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8bd3c422-2dfd-4f3c-9840-80142bd1a9bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8bd3c422-2dfd-4f3c-9840-80142bd1a9bf",
"name": "8bd3c422-2dfd-4f3c-9840-80142bd1a9bf",
"status": "Failed",
"startTime": "2026-11-04T15:36:49.872179674Z",
"endTime": "2026-11-04T16:05:58.807973445Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8bd3c422-2dfd-4f3c-9840-80142bd1a9bf
correlation ID: 40a4cd7e-ed59-441d-8424-9d083fa24b5e
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 1 · Nov 6: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T04:15:54Z"}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: <...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Deleting:; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x2e1c68c54ba0>:
failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-v5kvdw" resourcegroup="rg-np-version-upgrade-v5kvdw-szr6bkt4qz8b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da0f6f14-7cb5-4e48-af55-36eaa55c6018
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da0f6f14-7cb5-4e48-af55-36eaa55c6018",
"name": "da0f6f14-7cb5-4e48-af55-36eaa55c6018",
"status": "Failed",
"startTime": "2026-11-04T02:20:05.889371825Z",
"endTime": "2026-11-04T03:04:14.968301197Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: 1 of 4 machines are not ready\nMachine cluster-npupgrade-4-20-fjdqz-bpzbx: Deleting: \n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da0f6f14-7cb5-4e48-af55-36eaa55c6018
correlation ID: 09ca8682-0165-46dd-a6b7-402ea596ff1b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x2e1c68c54ba0>:
failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-v5kvdw" resourcegroup="rg-np-version-upgrade-v5kvdw-szr6bkt4qz8b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da0f6f14-7cb5-4e48-af55-36eaa55c6018
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da0f6f14-7cb5-4e48-af55-36eaa55c6018",
"name": "da0f6f14-7cb5-4e48-af55-36eaa55c6018",
"status": "Failed",
"startTime": "2026-11-04T02:20:05.889371825Z",
"endTime": "2026-11-04T03:04:14.968301197Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [hypershiftNodePool] hypershift NodePool condition AllMachinesReady is False: 1 of 4 machines are not ready\nMachine cluster-npupgrade-4-20-fjdqz-bpzbx: Deleting: \n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da0f6f14-7cb5-4e48-af55-36eaa55c6018
correlation ID: 09ca8682-0165-46dd-a6b7-402ea596ff1b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
VerifyConsoleLogin timed out after 15m0.01s: console proxy expected HTTP 200 without redirect, observed HTTP 401full failure pattern: VerifyConsoleLogin timed out after 15m0.01s: console proxy expected HTTP 200 without redirect, observed HTTP 401 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-4-21-z8tncn"
Unexpected error:
<*fmt.wrapError | 0x37fe5f17520>:
VerifyConsoleLogin timed out after 15m0.01s: console proxy expected HTTP 200 without redirect, observed HTTP 401
Dex namespace="console-9wj94tb"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-pz2zm condition="ContainersReady" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="Initialized" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="PodScheduled" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="Ready" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm container="dex" ready=true restarts=0; Pod/dex-676f8d46b9-pz2zm phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.01s: console proxy expected HTTP 200 without redirect, observed HTTP 401\nDex namespace=\"console-9wj94tb\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-pz2zm condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm container=\"dex\" ready=true restarts=0; Pod/dex-676f8d46b9-pz2zm phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x37fe76041f0>{
s: "console proxy expected HTTP 200 without redirect, observed HTTP 401",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-4-21-z8tncn"
Unexpected error:
<*fmt.wrapError | 0x37fe5f17520>:
VerifyConsoleLogin timed out after 15m0.01s: console proxy expected HTTP 200 without redirect, observed HTTP 401
Dex namespace="console-9wj94tb"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-pz2zm condition="ContainersReady" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="Initialized" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="PodScheduled" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm condition="Ready" status="True" reason=""; Pod/dex-676f8d46b9-pz2zm container="dex" ready=true restarts=0; Pod/dex-676f8d46b9-pz2zm phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.01s: console proxy expected HTTP 200 without redirect, observed HTTP 401\nDex namespace=\"console-9wj94tb\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-676f8d46b9-pz2zm condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-676f8d46b9-pz2zm container=\"dex\" ready=true restarts=0; Pod/dex-676f8d46b9-pz2zm phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x37fe76041f0>{
s: "console proxy expected HTTP 200 without redirect, observed HTTP 401",
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=False)full failure pattern: cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=False) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x2c979bd88000>:
context deadline exceeded
cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x2c979a3f9c20>{
s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=False)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x2c979bd88000>:
context deadline exceeded
cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x2c979a3f9c20>{
s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=False)",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)full failure pattern: cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x452712205e8>:
context deadline exceeded
cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x452719ab990>{
s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x452712205e8>:
context deadline exceeded
cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x452719ab990>{
s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-<id> cluster-np-1-kfkdx-<id>])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-<id> cluster-np-1-kfkdx-<id>]) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.001s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0x24b25d7d9480>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-8hfpb-rwtx8 cluster-np-1-kfkdx-kd9bj])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-8hfpb-rwtx8 cluster-np-1-kfkdx-kd9bj])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.001s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0x24b25d7d9480>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-8hfpb-rwtx8 cluster-np-1-kfkdx-kd9bj])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-8hfpb-rwtx8 cluster-np-1-kfkdx-kd9bj])",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-<id> cluster-np-1-sf6gt-ldlvj])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-<id> cluster-np-1-sf6gt-ldlvj]) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.000s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0xb68e34a4cd0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-5fcvx-nq72d cluster-np-1-sf6gt-ldlvj])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-5fcvx-nq72d cluster-np-1-sf6gt-ldlvj])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.000s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0xb68e34a4cd0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-5fcvx-nq72d cluster-np-1-sf6gt-ldlvj])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-5fcvx-nq72d cluster-np-1-sf6gt-ldlvj])",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-dnkj6-vjdfz cluster-np-1-pmcjc-<id>])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-dnkj6-vjdfz cluster-np-1-pmcjc-<id>]) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.001s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0x3042dcb1e7d0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-dnkj6-vjdfz cluster-np-1-pmcjc-7qczx])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-dnkj6-vjdfz cluster-np-1-pmcjc-7qczx])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.001s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0x3042dcb1e7d0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-dnkj6-vjdfz cluster-np-1-pmcjc-7qczx])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-dnkj6-vjdfz cluster-np-1-pmcjc-7qczx])",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-fqvdt-<id> cluster-np-1-xxrsg-<id>])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-fqvdt-<id> cluster-np-1-xxrsg-<id>]) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.003s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0x286b097edc70>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-fqvdt-pztn2 cluster-np-1-xxrsg-9x89t])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-fqvdt-pztn2 cluster-np-1-xxrsg-9x89t])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:823]: Timed out after 900.003s.
node pool np-1 never completed MI swap (day 2 attach + day 1 detach) for ACR pull
Expected success, but got an error:
<*errors.errorString | 0x286b097edc70>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-fqvdt-pztn2 cluster-np-1-xxrsg-9x89t])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-fqvdt-pztn2 cluster-np-1-xxrsg-9x89t])",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadlinefull failure pattern: failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x22b3d50895c0>:
context deadline exceeded
failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
{
errs: [
<context.deadlineExceededError>{},
<*fmt.wrapError | 0x22b3d5086fe0>{
msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x22b3d5086fc0>{
msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*errors.errorString | 0x22b3d6882480>{
s: "rate: Wait(n=1) would exceed context deadline",
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x22b3d50895c0>:
context deadline exceeded
failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
{
errs: [
<context.deadlineExceededError>{},
<*fmt.wrapError | 0x22b3d5086fe0>{
msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x22b3d5086fc0>{
msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*errors.errorString | 0x22b3d6882480>{
s: "rate: Wait(n=1) would exceed context deadline",
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
failed to list ClusterOperators: context deadline exceededfull failure pattern: failed to list ClusterOperators: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x81c23859b18>:
context deadline exceeded
failed to list ClusterOperators: context deadline exceeded
{
errs: [
<context.deadlineExceededError>{},
<*fmt.wrapError | 0x81c2368db80>{
msg: "failed to list ClusterOperators: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x81c23859b18>:
context deadline exceeded
failed to list ClusterOperators: context deadline exceeded
{
errs: [
<context.deadlineExceededError>{},
<*fmt.wrapError | 0x81c2368db80>{
msg: "failed to list ClusterOperators: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
node pool upgrade verification failed: cluster-nps-420-422-<id> (KubeletVersion v1full failure pattern: node pool upgrade verification failed: cluster-nps-420-422-<id> (KubeletVersion v1 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.56%1 of 39 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | |||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:494]: Timed out after 2700.000s.
Expected success, but got an error:
<*errors.errorString | 0x2058b9f45630>:
node pool upgrade verification failed: cluster-nps-420-422-rz7gd-f9p9g (KubeletVersion v1.33.13 has minor 33, expected 35 for OCP 4.22.21); nodes=[{"name":"cluster-nps-420-422-qtnxp-b2t7r","ready":true,"kubeletVersion":"v1.35.9","containerRuntimeVersion":"cri-o://1.35.10-3.rhaos4.22.git9383502.el9","imageCount":46},{"name":"cluster-nps-420-422-qtnxp-dm6w4","ready":true,"kubeletVersion":"v1.35.9","containerRuntimeVersion":"cri-o://1.35.10-3.rhaos4.22.git9383502.el9","imageCount":48},{"name":"cluster-nps-420-422-rz7gd-f9p9g","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-9.rhaos4.20.gitd6a41c4.el9","imageCount":49}]
{
s: "node pool upgrade verification failed: cluster-nps-420-422-rz7gd-f9p9g (KubeletVersion v1.33.13 has minor 33, expected 35 for OCP 4.22.21); nodes=[{\"name\":\"cluster-nps-420-422-qtnxp-b2t7r\",\"ready\":true,\"kubeletVersion\":\"v1.35.9\",\"containerRuntimeVersion\":\"cri-o://1.35.10-3.rhaos4.22.git9383502.el9\",\"imageCount\":46},{\"name\":\"cluster-nps-420-422-qtnxp-dm6w4\",\"ready\":true,\"kubeletVersion\":\"v1.35.9\",\"containerRuntimeVersion\":\"cri-o://1.35.10-3.rhaos4.22.git9383502.el9\",\"imageCount\":48},{\"name\":\"cluster-nps-420-422-rz7gd-f9p9g\",\"ready\":true,\"kubeletVersion\":\"v1.33.13\",\"containerRuntimeVersion\":\"cri-o://1.33.13-9.rhaos4.20.gitd6a41c4.el9\",\"imageCount\":49}]",
}
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:494]: Timed out after 2700.000s.
Expected success, but got an error:
<*errors.errorString | 0x2058b9f45630>:
node pool upgrade verification failed: cluster-nps-420-422-rz7gd-f9p9g (KubeletVersion v1.33.13 has minor 33, expected 35 for OCP 4.22.21); nodes=[{"name":"cluster-nps-420-422-qtnxp-b2t7r","ready":true,"kubeletVersion":"v1.35.9","containerRuntimeVersion":"cri-o://1.35.10-3.rhaos4.22.git9383502.el9","imageCount":46},{"name":"cluster-nps-420-422-qtnxp-dm6w4","ready":true,"kubeletVersion":"v1.35.9","containerRuntimeVersion":"cri-o://1.35.10-3.rhaos4.22.git9383502.el9","imageCount":48},{"name":"cluster-nps-420-422-rz7gd-f9p9g","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-9.rhaos4.20.gitd6a41c4.el9","imageCount":49}]
{
s: "node pool upgrade verification failed: cluster-nps-420-422-rz7gd-f9p9g (KubeletVersion v1.33.13 has minor 33, expected 35 for OCP 4.22.21); nodes=[{\"name\":\"cluster-nps-420-422-qtnxp-b2t7r\",\"ready\":true,\"kubeletVersion\":\"v1.35.9\",\"containerRuntimeVersion\":\"cri-o://1.35.10-3.rhaos4.22.git9383502.el9\",\"imageCount\":46},{\"name\":\"cluster-nps-420-422-qtnxp-dm6w4\",\"ready\":true,\"kubeletVersion\":\"v1.35.9\",\"containerRuntimeVersion\":\"cri-o://1.35.10-3.rhaos4.22.git9383502.el9\",\"imageCount\":48},{\"name\":\"cluster-nps-420-422-rz7gd-f9p9g\",\"ready\":true,\"kubeletVersion\":\"v1.33.13\",\"containerRuntimeVersion\":\"cri-o://1.33.13-9.rhaos4.20.gitd6a41c4.el9\",\"imageCount\":49}]",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
E2E Jobs
29
Success Rate
48.28%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.73.68% (14/19)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 6.90%2 of 29 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 12 · Nov 4: 6 · Nov 5: 0 · Nov 6: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x374759300170>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x374759300170>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xa5c19ed06d0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xa5c19ed06d0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2422075a690>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2422075a690>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (12)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpen...full failure pattern: ERROR CODE: GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 25 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
<*fmt.wrapError | 0x3b12782929c0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-gsj2rbf9gl9w" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
correlation ID: 9bc223c2-995a-427c-949e-62ca384432ff
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
<*fmt.wrapError | 0x3b12782929c0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-gsj2rbf9gl9w" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
correlation ID: 9bc223c2-995a-427c-949e-62ca384432ff
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x2b292c1c0168>:
resource group "rg-cordon-test-gr9c8pd6vnnm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gr9c8pd6vnnm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
correlation ID: c29bc14e-8e04-4ca1-9e28-44104dedab13
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x2b292c1c0168>:
resource group "rg-cordon-test-gr9c8pd6vnnm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gr9c8pd6vnnm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
correlation ID: c29bc14e-8e04-4ca1-9e28-44104dedab13
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0xe5b8ee500a8>:
resource group "clusternp128-br4ctdhmf9nj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-br4ctdhmf9nj" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
correlation ID: 93d49f6e-4f14-4abc-8522-d90532e9b6a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0xe5b8ee500a8>:
resource group "clusternp128-br4ctdhmf9nj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-br4ctdhmf9nj" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
correlation ID: 93d49f6e-4f14-4abc-8522-d90532e9b6a7
client request ID: unavailable
occurredContributing tests (25)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 12 · Nov 4: 6 · Nov 5: 0 · Nov 6: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e0c4e911a60>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e0c4e911a60>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x145f6ff3e730>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x145f6ff3e730>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x34f47ff7360>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x34f47ff7360>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 2 · Nov 6: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x28d3bbcb0540>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: 0fca0e5b-21cb-48a3-99c9-d833509498cb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x28d3bbcb0540>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: 0fca0e5b-21cb-48a3-99c9-d833509498cb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x350e2018c120>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: 086f246d-e2e2-426b-8dc9-754311467488
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x350e2018c120>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: 086f246d-e2e2-426b-8dc9-754311467488
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: ProviderError; detail message Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.; provider Microsoft.R...full failure pattern: ERROR CODE: ProviderError; detail message Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 2 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x31106c3400a8>:
resource group "rg-candidate-4-23-8g66dmqqdl2t": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-candidate-4-23-8g66dmqqdl2t": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: cdec6720-cba1-4de4-b58b-ba3824034d6e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x31106c3400a8>:
resource group "rg-candidate-4-23-8g66dmqqdl2t": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-candidate-4-23-8g66dmqqdl2t": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: cdec6720-cba1-4de4-b58b-ba3824034d6e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3509f2bfe0f0>:
resource group "autoscaling-cluster-s7nqf86dlmwk": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="autoscaling-cluster-s7nqf86dlmwk": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: c0db6013-b086-440f-864e-95153120e95f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3509f2bfe0f0>:
resource group "autoscaling-cluster-s7nqf86dlmwk": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="autoscaling-cluster-s7nqf86dlmwk": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: c0db6013-b086-440f-864e-95153120e95f
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x35318b3701a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-qgkvdgwhnvp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"name": "f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"status": "Failed",
"startTime": "2026-11-03T03:30:50.28074301Z",
"endTime": "2026-11-03T03:49:59.74254364Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
correlation ID: 414909e2-b5b8-45c7-b8ee-51a914e3f24f
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x35318b3701a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-qgkvdgwhnvp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"name": "f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"status": "Failed",
"startTime": "2026-11-03T03:30:50.28074301Z",
"endTime": "2026-11-03T03:49:59.74254364Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
correlation ID: 414909e2-b5b8-45c7-b8ee-51a914e3f24f
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: ConflictingConcurrentWriteNotAllowed; detail message The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later.; provider ...full failure pattern: ERROR CODE: ConflictingConcurrentWriteNotAllowed; detail message The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
<*exported.ResponseError | 0x3c8f7d214900>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ConflictingConcurrentWriteNotAllowed
--------------------------------------------------------------------------------
{
"error": {
"code": "ConflictingConcurrentWriteNotAllowed",
"message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
correlation ID: fbbd5005-0e31-417c-aa2b-b969b8f7ab06
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
<*exported.ResponseError | 0x3c8f7d214900>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ConflictingConcurrentWriteNotAllowed
--------------------------------------------------------------------------------
{
"error": {
"code": "ConflictingConcurrentWriteNotAllowed",
"message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
correlation ID: fbbd5005-0e31-417c-aa2b-b969b8f7ab06
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, cluster-image-registry-operator, cluster-network-operator, dns-operator, hosted-cluster-config-operator, ingress-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-vjvvwt6d4mf7/cluster-candidate-5-1-t5ntfs should provision
Unexpected error:
<*fmt.wrapError | 0x18ecee674de0>:
failed to create HCP cluster cluster-candidate-5-1-t5ntfs: failed waiting for cluster="cluster-candidate-5-1-t5ntfs" in resourcegroup="rg-candidate-5-1-vjvvwt6d4mf7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"name": "8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"status": "Failed",
"startTime": "2026-11-03T10:01:43.423478369Z",
"endTime": "2026-11-03T10:30:49.760962704Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, ingress-operator, catalog-operator, hosted-cluster-config-operator, cluster-network-operator, cluster-image-registry-operator, packageserver, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
correlation ID: d9c0e94e-e861-412f-a3b2-f93dbcd093b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-vjvvwt6d4mf7/cluster-candidate-5-1-t5ntfs should provision
Unexpected error:
<*fmt.wrapError | 0x18ecee674de0>:
failed to create HCP cluster cluster-candidate-5-1-t5ntfs: failed waiting for cluster="cluster-candidate-5-1-t5ntfs" in resourcegroup="rg-candidate-5-1-vjvvwt6d4mf7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"name": "8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"status": "Failed",
"startTime": "2026-11-03T10:01:43.423478369Z",
"endTime": "2026-11-03T10:30:49.760962704Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, ingress-operator, catalog-operator, hosted-cluster-config-operator, cluster-network-operator, cluster-image-registry-operator, packageserver, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
correlation ID: d9c0e94e-e861-412f-a3b2-f93dbcd093b1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x2486b0990c0>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fhxl6psjw7xx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9",
"name": "23e1c59e-d85b-40a6-b315-78ca430171e9",
"status": "Failed",
"startTime": "2026-11-03T03:31:25.980900033Z",
"endTime": "2026-11-03T03:50:29.670327156Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
correlation ID: 7d147fcf-e938-458b-891d-77c7c176f302
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x2486b0990c0>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fhxl6psjw7xx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9",
"name": "23e1c59e-d85b-40a6-b315-78ca430171e9",
"status": "Failed",
"startTime": "2026-11-03T03:31:25.980900033Z",
"endTime": "2026-11-03T03:50:29.670327156Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
correlation ID: 7d147fcf-e938-458b-891d-77c7c176f302
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidResource; detail message ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions.; provider Mic...full failure pattern: ERROR CODE: InvalidResource; detail message ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x2e79db1b2e0>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-22mskvfrsnk8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InvalidResource
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936",
"name": "f4b8761b-d251-4636-b586-532f30b02936",
"status": "Failed",
"startTime": "2026-11-03T07:12:43.456403262Z",
"endTime": "2026-11-03T07:23:57.559738554Z",
"error": {
"code": "InvalidResource",
"message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
correlation ID: 3507dcf7-3522-4e9c-bdf1-375d292a8d55
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x2e79db1b2e0>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-22mskvfrsnk8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InvalidResource
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936",
"name": "f4b8761b-d251-4636-b586-532f30b02936",
"status": "Failed",
"startTime": "2026-11-03T07:12:43.456403262Z",
"endTime": "2026-11-03T07:23:57.559738554Z",
"error": {
"code": "InvalidResource",
"message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
correlation ID: 3507dcf7-3522-4e9c-bdf1-375d292a8d55
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-02T09:54:16Z"}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
E2E Jobs
50
Success Rate
58.00%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.63.04% (29/46)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 90.91% | 11 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 90.91% | 11 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 90.91% | 11 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 90.91% | 11 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 90.91% | 11 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 6.00%3 of 50 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 2 · Nov 5: 1 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x163093bdc660>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-x4cn4q46hs7r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/16975619-9ef1-44ec-89ab-5ab29fd11e3b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/16975619-9ef1-44ec-89ab-5ab29fd11e3b",
"name": "16975619-9ef1-44ec-89ab-5ab29fd11e3b",
"status": "Failed",
"startTime": "2026-11-05T03:15:33.316031452Z",
"endTime": "2026-11-05T03:34:35.654618083Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/16975619-9ef1-44ec-89ab-5ab29fd11e3b
correlation ID: b37e6a2d-04ad-4043-89a6-da9a36474d78
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x163093bdc660>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-x4cn4q46hs7r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/16975619-9ef1-44ec-89ab-5ab29fd11e3b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/16975619-9ef1-44ec-89ab-5ab29fd11e3b",
"name": "16975619-9ef1-44ec-89ab-5ab29fd11e3b",
"status": "Failed",
"startTime": "2026-11-05T03:15:33.316031452Z",
"endTime": "2026-11-05T03:34:35.654618083Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/16975619-9ef1-44ec-89ab-5ab29fd11e3b
correlation ID: b37e6a2d-04ad-4043-89a6-da9a36474d78
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x15783fba4380>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-5fj5jpqw8b8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b061f7f8-0efd-4712-86c7-efd9b752b6b7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b061f7f8-0efd-4712-86c7-efd9b752b6b7",
"name": "b061f7f8-0efd-4712-86c7-efd9b752b6b7",
"status": "Failed",
"startTime": "2026-11-05T00:43:54.451174917Z",
"endTime": "2026-11-05T01:02:58.723032504Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b061f7f8-0efd-4712-86c7-efd9b752b6b7
correlation ID: 60a087a7-1ba6-45fe-af29-d857f1006161
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x15783fba4380>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-5fj5jpqw8b8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b061f7f8-0efd-4712-86c7-efd9b752b6b7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b061f7f8-0efd-4712-86c7-efd9b752b6b7",
"name": "b061f7f8-0efd-4712-86c7-efd9b752b6b7",
"status": "Failed",
"startTime": "2026-11-05T00:43:54.451174917Z",
"endTime": "2026-11-05T01:02:58.723032504Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b061f7f8-0efd-4712-86c7-efd9b752b6b7
correlation ID: 60a087a7-1ba6-45fe-af29-d857f1006161
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x1048e03529e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-dmm9cjvqmfqq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3feb9df5-7da6-4159-b554-497d5bd164c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3feb9df5-7da6-4159-b554-497d5bd164c0",
"name": "3feb9df5-7da6-4159-b554-497d5bd164c0",
"status": "Failed",
"startTime": "2026-11-04T16:55:03.51721993Z",
"endTime": "2026-11-04T17:14:08.989788377Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3feb9df5-7da6-4159-b554-497d5bd164c0
correlation ID: 4938b726-2fd4-4be8-86c0-2140125b6c9b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x1048e03529e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-dmm9cjvqmfqq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3feb9df5-7da6-4159-b554-497d5bd164c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3feb9df5-7da6-4159-b554-497d5bd164c0",
"name": "3feb9df5-7da6-4159-b554-497d5bd164c0",
"status": "Failed",
"startTime": "2026-11-04T16:55:03.51721993Z",
"endTime": "2026-11-04T17:14:08.989788377Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3feb9df5-7da6-4159-b554-497d5bd164c0
correlation ID: 4938b726-2fd4-4be8-86c0-2140125b6c9b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 6.00%3 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 12 · Nov 4: 6 · Nov 5: 0 · Nov 6: 0 | DEV, INT, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x17f2c21d05e0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x17f2c21d05e0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x22eb02809c60>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x22eb02809c60>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3716a53c4e80>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3716a53c4e80>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (12)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | unknown | 3 | 6.00%3 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 1 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | DEV, INT, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-04T19:11:01Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-03T23:07:45Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-02T02:07:08Z"}Contributing tests (3)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 4.00%2 of 50 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 · Nov 5: 1 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-4l8tt7" is viable
Unexpected error:
<*errors.joinError | 0x538c7490030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x538c6e1c1a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x538c6e1c180>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x538c70ecdc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x538c669edc0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x538c669ebc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x538c6e7a6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-4l8tt7" is viable
Unexpected error:
<*errors.joinError | 0x538c7490030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x538c6e1c1a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x538c6e1c180>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x538c70ecdc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x538c669edc0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x538c669ebc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x538c6e7a6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-dfv4zc" is viable
Unexpected error:
<*errors.joinError | 0x1851dc4f1470>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1851dc18a360>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1851dc18a340>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1851dc0e60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1851dbd57e20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1851dbd57de0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1851dbe77180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-dfv4zc" is viable
Unexpected error:
<*errors.joinError | 0x1851dc4f1470>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1851dc18a360>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1851dc18a340>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1851dc0e60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1851dbd57e20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1851dbd57de0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1851dbe77180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 2 | 4.00%2 of 50 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 1 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 1 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:199]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0xdd566cf84c8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0xdd567282d20>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0xdd567282d00>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0xdd5670ae5a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:199]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0xdd566cf84c8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0xdd567282d20>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0xdd567282d00>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0xdd5670ae5a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-m6nrpn" is viable
Unexpected error:
<*errors.joinError | 0x2a7e709ee378>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2a7e705ff140>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2a7e705ff120>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2a7e7061a640>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-m6nrpn" is viable
Unexpected error:
<*errors.joinError | 0x2a7e709ee378>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2a7e705ff140>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2a7e705ff120>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2a7e7061a640>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 69 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x357a2fe5f040>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-csp4tk4h8qs9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: 49abcf5f-88e9-49d5-b8c2-5f1d082bc74c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x357a2fe5f040>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-csp4tk4h8qs9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: 49abcf5f-88e9-49d5-b8c2-5f1d082bc74c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-mcx8z4" with version 4.22.20 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3e43e6d40140>:
failed to create HCP cluster cluster-zstream-4-22-mcx8z4: failed starting cluster creation "cluster-zstream-4-22-mcx8z4" in resourcegroup="rg-zstream-upgrade-4-22-s6jpsvnr8lwq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
correlation ID: a7c5e206-75d9-4088-8da5-8a945905e74d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-mcx8z4" with version 4.22.20 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3e43e6d40140>:
failed to create HCP cluster cluster-zstream-4-22-mcx8z4: failed starting cluster creation "cluster-zstream-4-22-mcx8z4" in resourcegroup="rg-zstream-upgrade-4-22-s6jpsvnr8lwq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
correlation ID: a7c5e206-75d9-4088-8da5-8a945905e74d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xeef026fef40>:
failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-hsxdls4m5cf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
correlation ID: 0d18fc06-8f55-4eae-9e49-cbd4e9f7546f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xeef026fef40>:
failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-hsxdls4m5cf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
correlation ID: 0d18fc06-8f55-4eae-9e49-cbd4e9f7546f
client request ID: unavailable
occurredContributing tests (69)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 4 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3a3d164f6288>:
resource group "rg-zstream-upgrade-4-20-dctkglbphsrd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-rdqr86" in resourcegroup="rg-zstream-upgrade-4-20-dctkglbphsrd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eac4dbf3-b99d-4ff5-b55a-c5cb400107ba
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eac4dbf3-b99d-4ff5-b55a-c5cb400107ba",
"name": "eac4dbf3-b99d-4ff5-b55a-c5cb400107ba",
"status": "Failed",
"startTime": "2026-11-04T20:46:37.24040578Z",
"endTime": "2026-11-04T21:10:42.611007312Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tska6a736668pas8l1dsqki7si2iboj still exists (deletion dispatched at 2026-11-04T20:46:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eac4dbf3-b99d-4ff5-b55a-c5cb400107ba
correlation ID: 5f525aa7-22c6-4b79-ab5e-6d15911e1937
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3a3d164f6288>:
resource group "rg-zstream-upgrade-4-20-dctkglbphsrd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-rdqr86" in resourcegroup="rg-zstream-upgrade-4-20-dctkglbphsrd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eac4dbf3-b99d-4ff5-b55a-c5cb400107ba
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eac4dbf3-b99d-4ff5-b55a-c5cb400107ba",
"name": "eac4dbf3-b99d-4ff5-b55a-c5cb400107ba",
"status": "Failed",
"startTime": "2026-11-04T20:46:37.24040578Z",
"endTime": "2026-11-04T21:10:42.611007312Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tska6a736668pas8l1dsqki7si2iboj still exists (deletion dispatched at 2026-11-04T20:46:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/eac4dbf3-b99d-4ff5-b55a-c5cb400107ba
correlation ID: 5f525aa7-22c6-4b79-ab5e-6d15911e1937
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3ddb886742e8>:
resource group "rg-candidate-5-1-qxsgtfwcgcpz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-6klmdv" in resourcegroup="rg-candidate-5-1-qxsgtfwcgcpz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8b354ea-5b88-48e7-adad-6ff810bdf73e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8b354ea-5b88-48e7-adad-6ff810bdf73e",
"name": "b8b354ea-5b88-48e7-adad-6ff810bdf73e",
"status": "Failed",
"startTime": "2026-11-04T20:50:41.701632076Z",
"endTime": "2026-11-04T21:14:42.598224003Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tsk9i3ooiq6k9mpn4m8nrs500hd4tea still exists (deletion dispatched at 2026-11-04T20:50:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8b354ea-5b88-48e7-adad-6ff810bdf73e
correlation ID: 9004b7cf-ce6f-4089-bc34-c9c03779c4fe
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3ddb886742e8>:
resource group "rg-candidate-5-1-qxsgtfwcgcpz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-6klmdv" in resourcegroup="rg-candidate-5-1-qxsgtfwcgcpz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8b354ea-5b88-48e7-adad-6ff810bdf73e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8b354ea-5b88-48e7-adad-6ff810bdf73e",
"name": "b8b354ea-5b88-48e7-adad-6ff810bdf73e",
"status": "Failed",
"startTime": "2026-11-04T20:50:41.701632076Z",
"endTime": "2026-11-04T21:14:42.598224003Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tsk9i3ooiq6k9mpn4m8nrs500hd4tea still exists (deletion dispatched at 2026-11-04T20:50:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8b354ea-5b88-48e7-adad-6ff810bdf73e
correlation ID: 9004b7cf-ce6f-4089-bc34-c9c03779c4fe
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3fde3d86c318>:
resource group "rg-cp-ystream-upgrade-5-0-jr49hlksm7hz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-272rf4" in resourcegroup="rg-cp-ystream-upgrade-5-0-jr49hlksm7hz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9603b3c2-96bc-4fee-98c7-351436279fc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9603b3c2-96bc-4fee-98c7-351436279fc7",
"name": "9603b3c2-96bc-4fee-98c7-351436279fc7",
"status": "Failed",
"startTime": "2026-11-04T20:48:09.059861424Z",
"endTime": "2026-11-04T21:12:12.6008863Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tskc3ocaso2auk6pjo9sb18lg90e87u still exists (deletion dispatched at 2026-11-04T20:48:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9603b3c2-96bc-4fee-98c7-351436279fc7
correlation ID: 7fb39f32-4781-4007-a687-d687cf44201c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3fde3d86c318>:
resource group "rg-cp-ystream-upgrade-5-0-jr49hlksm7hz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-272rf4" in resourcegroup="rg-cp-ystream-upgrade-5-0-jr49hlksm7hz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9603b3c2-96bc-4fee-98c7-351436279fc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9603b3c2-96bc-4fee-98c7-351436279fc7",
"name": "9603b3c2-96bc-4fee-98c7-351436279fc7",
"status": "Failed",
"startTime": "2026-11-04T20:48:09.059861424Z",
"endTime": "2026-11-04T21:12:12.6008863Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tskc3ocaso2auk6pjo9sb18lg90e87u still exists (deletion dispatched at 2026-11-04T20:48:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9603b3c2-96bc-4fee-98c7-351436279fc7
correlation ID: 7fb39f32-4781-4007-a687-d687cf44201c
client request ID: unavailable
occurredContributing tests (4)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 2 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x2cd540df6288>:
resource group "np-autoscaling-7mgtwmrntx9q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-7mgtwmrntx9q" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f5fd39eb-109e-4102-983a-2b9360a38a7c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f5fd39eb-109e-4102-983a-2b9360a38a7c",
"name": "f5fd39eb-109e-4102-983a-2b9360a38a7c",
"status": "Failed",
"startTime": "2026-11-04T21:16:21.832982583Z",
"endTime": "2026-11-04T21:40:25.60036215Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tskl0sehh8vs0465jcd240skjg7lec5 still exists (deletion dispatched at 2026-11-04T21:16:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f5fd39eb-109e-4102-983a-2b9360a38a7c
correlation ID: 476c62a1-a2c2-42f1-8f70-862282725baf
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x2cd540df6288>:
resource group "np-autoscaling-7mgtwmrntx9q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-7mgtwmrntx9q" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f5fd39eb-109e-4102-983a-2b9360a38a7c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f5fd39eb-109e-4102-983a-2b9360a38a7c",
"name": "f5fd39eb-109e-4102-983a-2b9360a38a7c",
"status": "Failed",
"startTime": "2026-11-04T21:16:21.832982583Z",
"endTime": "2026-11-04T21:40:25.60036215Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tskl0sehh8vs0465jcd240skjg7lec5 still exists (deletion dispatched at 2026-11-04T21:16:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f5fd39eb-109e-4102-983a-2b9360a38a7c
correlation ID: 476c62a1-a2c2-42f1-8f70-862282725baf
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x19f851d257b8>:
resource group "full-priv-kv-gzjpqjw9wfc9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="full-priv-kv" in resourcegroup="full-priv-kv-gzjpqjw9wfc9" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f42d96ea-ecf8-4647-ba80-a0b740847e3d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f42d96ea-ecf8-4647-ba80-a0b740847e3d",
"name": "f42d96ea-ecf8-4647-ba80-a0b740847e3d",
"status": "Failed",
"startTime": "2026-11-04T20:20:04.012581574Z",
"endTime": "2026-11-04T20:44:15.273167074Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tsju2q0magp3kjh2e58as6c27u20eq3 still exists (deletion dispatched at 2026-11-04T20:20:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f42d96ea-ecf8-4647-ba80-a0b740847e3d
correlation ID: 9bd12494-7a3e-45b0-8dde-85e33ff3e00c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x19f851d257b8>:
resource group "full-priv-kv-gzjpqjw9wfc9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="full-priv-kv" in resourcegroup="full-priv-kv-gzjpqjw9wfc9" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f42d96ea-ecf8-4647-ba80-a0b740847e3d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f42d96ea-ecf8-4647-ba80-a0b740847e3d",
"name": "f42d96ea-ecf8-4647-ba80-a0b740847e3d",
"status": "Failed",
"startTime": "2026-11-04T20:20:04.012581574Z",
"endTime": "2026-11-04T20:44:15.273167074Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tsju2q0magp3kjh2e58as6c27u20eq3 still exists (deletion dispatched at 2026-11-04T20:20:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f42d96ea-ecf8-4647-ba80-a0b740847e3d
correlation ID: 9bd12494-7a3e-45b0-8dde-85e33ff3e00c
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 2 · Nov 6: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x3b01c514c480>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 9fac9949-3a2c-41de-bbe1-9a4554e5277e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x3b01c514c480>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 9fac9949-3a2c-41de-bbe1-9a4554e5277e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0xa26466acd20>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 1fd8da6c-a256-4f78-bfa6-ea613dc0906d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0xa26466acd20>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 1fd8da6c-a256-4f78-bfa6-ea613dc0906d
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft....full failure pattern: ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/<resource>' over scope '/subscriptions/<subscription>/resourceG... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x23a5ce5e9880>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-wvt46n2qkd8m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6",
"name": "b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6",
"status": "Failed",
"startTime": "2026-11-04T20:56:10.143982064Z",
"endTime": "2026-11-04T21:15:12.4205027Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-ephemeral-np-mpvdf-snrw2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-ephemeral-np-mpvdf-x8lt7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-ephemeral-np-mpvdf-snrw2: Failed: virtualmachine failed to create or update. err: failed to reapply VM ephemeral-osdisk-wvt46n2qkd8m--managed/cluster-ephemeral-np-mpvdf-snrw2: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-wvt46n2qkd8m--managed/providers/Microsoft.Compute/virtualMachines/cluster-ephemeral-np-mpvdf-snrw2/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'da4bd657-eb57-415b-9650-95aec9a48303' with object id 'd0d6a08a-10b4-497d-8acb-72ae5c5b5fca' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceG... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6
correlation ID: 30985f10-84c6-4031-a997-0a7d7c89e2a4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x23a5ce5e9880>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-wvt46n2qkd8m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6",
"name": "b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6",
"status": "Failed",
"startTime": "2026-11-04T20:56:10.143982064Z",
"endTime": "2026-11-04T21:15:12.4205027Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-ephemeral-np-mpvdf-snrw2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-ephemeral-np-mpvdf-x8lt7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-ephemeral-np-mpvdf-snrw2: Failed: virtualmachine failed to create or update. err: failed to reapply VM ephemeral-osdisk-wvt46n2qkd8m--managed/cluster-ephemeral-np-mpvdf-snrw2: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-wvt46n2qkd8m--managed/providers/Microsoft.Compute/virtualMachines/cluster-ephemeral-np-mpvdf-snrw2/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'da4bd657-eb57-415b-9650-95aec9a48303' with object id 'd0d6a08a-10b4-497d-8acb-72ae5c5b5fca' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceG... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b25a8dfd-08d3-4c16-aa9d-a0c7a8c2beb6
correlation ID: 30985f10-84c6-4031-a997-0a7d7c89e2a4
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft....full failure pattern: ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/<resource>' over scope '/subscriptions/<subscription>/resourceGroups... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
<*fmt.wrapError | 0x2de807269680>:
failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-zt8k7v98swfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/643a104b-a92a-4b73-8872-fbfdb53d5ec3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/643a104b-a92a-4b73-8872-fbfdb53d5ec3",
"name": "643a104b-a92a-4b73-8872-fbfdb53d5ec3",
"status": "Failed",
"startTime": "2026-11-04T20:38:32.864650088Z",
"endTime": "2026-11-04T20:57:32.516376066Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-ea-np-1-nczzz-mcmgd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-ea-np-1-nczzz-mcmgd: Failed: virtualmachine failed to create or update. err: failed to reapply VM external-auth-cluster-zt8k7v98swfn--managed/cluster-ea-np-1-nczzz-mcmgd: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-zt8k7v98swfn--managed/providers/Microsoft.Compute/virtualMachines/cluster-ea-np-1-nczzz-mcmgd/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'bc9647d7-bc6e-44a8-82a6-fca0dbf71333' with object id '60bffd2e-323f-400f-a839-599e2f71181f' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/643a104b-a92a-4b73-8872-fbfdb53d5ec3
correlation ID: 85e4a406-4df5-4452-b535-c67c29f2b08e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
<*fmt.wrapError | 0x2de807269680>:
failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-zt8k7v98swfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/643a104b-a92a-4b73-8872-fbfdb53d5ec3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/643a104b-a92a-4b73-8872-fbfdb53d5ec3",
"name": "643a104b-a92a-4b73-8872-fbfdb53d5ec3",
"status": "Failed",
"startTime": "2026-11-04T20:38:32.864650088Z",
"endTime": "2026-11-04T20:57:32.516376066Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-ea-np-1-nczzz-mcmgd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-ea-np-1-nczzz-mcmgd: Failed: virtualmachine failed to create or update. err: failed to reapply VM external-auth-cluster-zt8k7v98swfn--managed/cluster-ea-np-1-nczzz-mcmgd: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-zt8k7v98swfn--managed/providers/Microsoft.Compute/virtualMachines/cluster-ea-np-1-nczzz-mcmgd/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'bc9647d7-bc6e-44a8-82a6-fca0dbf71333' with object id '60bffd2e-323f-400f-a839-599e2f71181f' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/643a104b-a92a-4b73-8872-fbfdb53d5ec3
correlation ID: 85e4a406-4df5-4452-b535-c67c29f2b08e
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft....full failure pattern: ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/<resource>' over scope '/subscriptions/<subscription>/resourceGroups/cx-rg-cluster-s78c2tgdgl... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x33201089abc0>:
failed to create NodePool np-2: failed waiting for nodepool="np-2" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-s78c2tgdgl72" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/aec9dc7f-b6cb-4f34-8dad-15123f04b3a3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/aec9dc7f-b6cb-4f34-8dad-15123f04b3a3",
"name": "aec9dc7f-b6cb-4f34-8dad-15123f04b3a3",
"status": "Failed",
"startTime": "2026-11-04T12:39:43.390853829Z",
"endTime": "2026-11-04T12:58:52.711854196Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine cluster-np-2-grlbb-jgpg8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine cluster-np-2-grlbb-jgpg8: Failed: virtualmachine failed to create or update. err: failed to reapply VM cx-rg-cluster-s78c2tgdgl72--managed/cluster-np-2-grlbb-jgpg8: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-s78c2tgdgl72--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-2-grlbb-jgpg8/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'cdac37c1-20b3-4638-9cd5-33a73ef6c998' with object id 'df4916dc-1fc4-4052-a260-f098bfa83915' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-s78c2tgdgl... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/aec9dc7f-b6cb-4f34-8dad-15123f04b3a3
correlation ID: 5ef1a456-002a-44bc-9f95-6349c8d4c63f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x33201089abc0>:
failed to create NodePool np-2: failed waiting for nodepool="np-2" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-s78c2tgdgl72" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/aec9dc7f-b6cb-4f34-8dad-15123f04b3a3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/aec9dc7f-b6cb-4f34-8dad-15123f04b3a3",
"name": "aec9dc7f-b6cb-4f34-8dad-15123f04b3a3",
"status": "Failed",
"startTime": "2026-11-04T12:39:43.390853829Z",
"endTime": "2026-11-04T12:58:52.711854196Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine cluster-np-2-grlbb-jgpg8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine cluster-np-2-grlbb-jgpg8: Failed: virtualmachine failed to create or update. err: failed to reapply VM cx-rg-cluster-s78c2tgdgl72--managed/cluster-np-2-grlbb-jgpg8: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-s78c2tgdgl72--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-2-grlbb-jgpg8/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'cdac37c1-20b3-4638-9cd5-33a73ef6c998' with object id 'df4916dc-1fc4-4052-a260-f098bfa83915' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-s78c2tgdgl... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/aec9dc7f-b6cb-4f34-8dad-15123f04b3a3
correlation ID: 5ef1a456-002a-44bc-9f95-6349c8d4c63f
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0x7b35bcbd860>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2rfl69v22jl4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/10ae7a10-7399-4e3e-911b-6d6ca5d8202e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/10ae7a10-7399-4e3e-911b-6d6ca5d8202e",
"name": "10ae7a10-7399-4e3e-911b-6d6ca5d8202e",
"status": "Failed",
"startTime": "2026-11-04T20:16:49.016137203Z",
"endTime": "2026-11-04T20:35:53.076314768Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-mknpk-5qrlk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-mknpk-5qrlk: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/10ae7a10-7399-4e3e-911b-6d6ca5d8202e
correlation ID: a157f5c2-0761-4296-9fe2-159f5d44fb17
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0x7b35bcbd860>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2rfl69v22jl4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/10ae7a10-7399-4e3e-911b-6d6ca5d8202e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/10ae7a10-7399-4e3e-911b-6d6ca5d8202e",
"name": "10ae7a10-7399-4e3e-911b-6d6ca5d8202e",
"status": "Failed",
"startTime": "2026-11-04T20:16:49.016137203Z",
"endTime": "2026-11-04T20:35:53.076314768Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-mknpk-5qrlk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-mknpk-5qrlk: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/10ae7a10-7399-4e3e-911b-6d6ca5d8202e
correlation ID: a157f5c2-0761-4296-9fe2-159f5d44fb17
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource oidc-wi-f64s557pf5fc--managed/cluster-oidc-wi-np-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x20875e82e3c0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-f64s557pf5fc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/43be61ab-954f-4371-b2a0-72e9d710db26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/43be61ab-954f-4371-b2a0-72e9d710db26",
"name": "43be61ab-954f-4371-b2a0-72e9d710db26",
"status": "Failed",
"startTime": "2026-11-04T21:05:27.559806934Z",
"endTime": "2026-11-04T21:24:32.458898915Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-oidc-wi-np-8qwwn-8d6rg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-oidc-wi-np-8qwwn-8d6rg: Failed: virtualmachine failed to create or update. err: failed to create or update resource oidc-wi-f64s557pf5fc--managed/cluster-oidc-wi-np-8qwwn-8d6rg (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/43be61ab-954f-4371-b2a0-72e9d710db26
correlation ID: 38dfb016-0504-40c5-918f-3b44d4947afa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x20875e82e3c0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-f64s557pf5fc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/43be61ab-954f-4371-b2a0-72e9d710db26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/43be61ab-954f-4371-b2a0-72e9d710db26",
"name": "43be61ab-954f-4371-b2a0-72e9d710db26",
"status": "Failed",
"startTime": "2026-11-04T21:05:27.559806934Z",
"endTime": "2026-11-04T21:24:32.458898915Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-oidc-wi-np-8qwwn-8d6rg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-oidc-wi-np-8qwwn-8d6rg: Failed: virtualmachine failed to create or update. err: failed to create or update resource oidc-wi-f64s557pf5fc--managed/cluster-oidc-wi-np-8qwwn-8d6rg (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/43be61ab-954f-4371-b2a0-72e9d710db26
correlation ID: 38dfb016-0504-40c5-918f-3b44d4947afa
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-negative-tests-s84jzxrprbch--managed/cluster-np-1-72wqq-bxmbb (service: virtualmachine): context deadline exceeded Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-negative-tests-s84jzxrprbch--managed/cluster-np-1-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x2fc8aec032c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-s84jzxrprbch" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e92d32d7-60d3-4bb4-b6c9-0c60580b43fe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e92d32d7-60d3-4bb4-b6c9-0c60580b43fe",
"name": "e92d32d7-60d3-4bb4-b6c9-0c60580b43fe",
"status": "Failed",
"startTime": "2026-11-04T21:06:53.901624447Z",
"endTime": "2026-11-04T21:26:02.465424343Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-72wqq-bxmbb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-72wqq-m2q2d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-72wqq-bxmbb: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-negative-tests-s84jzxrprbch--managed/cluster-np-1-72wqq-bxmbb (service: virtualmachine): context deadline exceeded\nMachine cluster-np-1-72wqq-m2q2d: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-negative-tests-s84jzxrprbch--managed/cluster-np-1-72wqq-m2q2d (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e92d32d7-60d3-4bb4-b6c9-0c60580b43fe
correlation ID: a74d6691-2190-4192-a33f-9a02db09e2fa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x2fc8aec032c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-s84jzxrprbch" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e92d32d7-60d3-4bb4-b6c9-0c60580b43fe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e92d32d7-60d3-4bb4-b6c9-0c60580b43fe",
"name": "e92d32d7-60d3-4bb4-b6c9-0c60580b43fe",
"status": "Failed",
"startTime": "2026-11-04T21:06:53.901624447Z",
"endTime": "2026-11-04T21:26:02.465424343Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-72wqq-bxmbb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-72wqq-m2q2d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-72wqq-bxmbb: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-negative-tests-s84jzxrprbch--managed/cluster-np-1-72wqq-bxmbb (service: virtualmachine): context deadline exceeded\nMachine cluster-np-1-72wqq-m2q2d: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-negative-tests-s84jzxrprbch--managed/cluster-np-1-72wqq-m2q2d (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e92d32d7-60d3-4bb4-b6c9-0c60580b43fe
correlation ID: a74d6691-2190-4192-a33f-9a02db09e2fa
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-downgrade-wsmzxs-d8fvlb4vggq4-np-dg-wsmzxs--managed/cluster-npdg-4-21-npvmn-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
<*fmt.wrapError | 0x36e043ab0820>:
failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-wsmzxs" in resourcegroup="rg-np-downgrade-wsmzxs-d8fvlb4vggq4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/698d35ef-8188-46ae-b011-2d8f2573fb5b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/698d35ef-8188-46ae-b011-2d8f2573fb5b",
"name": "698d35ef-8188-46ae-b011-2d8f2573fb5b",
"status": "Failed",
"startTime": "2026-11-04T21:03:10.692151241Z",
"endTime": "2026-11-04T21:22:12.413497817Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npdg-4-21-npvmn-n4pr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npdg-4-21-npvmn-n4pr6: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-downgrade-wsmzxs-d8fvlb4vggq4-np-dg-wsmzxs--managed/cluster-npdg-4-21-npvmn-n4pr6 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/698d35ef-8188-46ae-b011-2d8f2573fb5b
correlation ID: 26a15889-d3f9-43eb-8e28-9e1131294f03
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
<*fmt.wrapError | 0x36e043ab0820>:
failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-wsmzxs" in resourcegroup="rg-np-downgrade-wsmzxs-d8fvlb4vggq4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/698d35ef-8188-46ae-b011-2d8f2573fb5b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/698d35ef-8188-46ae-b011-2d8f2573fb5b",
"name": "698d35ef-8188-46ae-b011-2d8f2573fb5b",
"status": "Failed",
"startTime": "2026-11-04T21:03:10.692151241Z",
"endTime": "2026-11-04T21:22:12.413497817Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npdg-4-21-npvmn-n4pr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npdg-4-21-npvmn-n4pr6: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-downgrade-wsmzxs-d8fvlb4vggq4-np-dg-wsmzxs--managed/cluster-npdg-4-21-npvmn-n4pr6 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/698d35ef-8188-46ae-b011-2d8f2573fb5b
correlation ID: 26a15889-d3f9-43eb-8e28-9e1131294f03
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-noedge-qd49pb-gwnw2h6mdl5l-np-ne-qd49pb--managed/cluster-npnoedge-4-20-ztsfp-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x27a6883e8de0>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-qd49pb" in resourcegroup="rg-np-noedge-qd49pb-gwnw2h6mdl5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5adcbebf-676d-4cf0-b225-c63e9dc8c201
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5adcbebf-676d-4cf0-b225-c63e9dc8c201",
"name": "5adcbebf-676d-4cf0-b225-c63e9dc8c201",
"status": "Failed",
"startTime": "2026-11-04T21:04:01.867875072Z",
"endTime": "2026-11-04T21:23:02.42304438Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-ztsfp-x9jpn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-ztsfp-x9jpn: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-noedge-qd49pb-gwnw2h6mdl5l-np-ne-qd49pb--managed/cluster-npnoedge-4-20-ztsfp-x9jpn (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5adcbebf-676d-4cf0-b225-c63e9dc8c201
correlation ID: 910a9c95-3bdd-4df1-a24c-1f5e71f93f07
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x27a6883e8de0>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-qd49pb" in resourcegroup="rg-np-noedge-qd49pb-gwnw2h6mdl5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5adcbebf-676d-4cf0-b225-c63e9dc8c201
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5adcbebf-676d-4cf0-b225-c63e9dc8c201",
"name": "5adcbebf-676d-4cf0-b225-c63e9dc8c201",
"status": "Failed",
"startTime": "2026-11-04T21:04:01.867875072Z",
"endTime": "2026-11-04T21:23:02.42304438Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-ztsfp-x9jpn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-ztsfp-x9jpn: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-noedge-qd49pb-gwnw2h6mdl5l-np-ne-qd49pb--managed/cluster-npnoedge-4-20-ztsfp-x9jpn (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5adcbebf-676d-4cf0-b225-c63e9dc8c201
correlation ID: 910a9c95-3bdd-4df1-a24c-1f5e71f93f07
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-skip-minor-4gqpsc-r6vc8cv4qnr7-np-sm-4gqpsc--managed/cluster-nps-420-422-dhdgt-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x509bf97920>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-4gqpsc" in resourcegroup="rg-np-skip-minor-4gqpsc-r6vc8cv4qnr7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2",
"name": "8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2",
"status": "Failed",
"startTime": "2026-11-04T21:03:30.202147603Z",
"endTime": "2026-11-04T21:22:32.435745891Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-nps-420-422-dhdgt-5wgxh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-nps-420-422-dhdgt-5wgxh: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-skip-minor-4gqpsc-r6vc8cv4qnr7-np-sm-4gqpsc--managed/cluster-nps-420-422-dhdgt-5wgxh (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2
correlation ID: aa1fc101-b074-42ef-8993-73eda65dca85
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x509bf97920>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-4gqpsc" in resourcegroup="rg-np-skip-minor-4gqpsc-r6vc8cv4qnr7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2",
"name": "8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2",
"status": "Failed",
"startTime": "2026-11-04T21:03:30.202147603Z",
"endTime": "2026-11-04T21:22:32.435745891Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-nps-420-422-dhdgt-5wgxh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-nps-420-422-dhdgt-5wgxh: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-skip-minor-4gqpsc-r6vc8cv4qnr7-np-sm-4gqpsc--managed/cluster-nps-420-422-dhdgt-5wgxh (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8eb6ba0e-5d36-4df6-a39b-30753b7f0fb2
correlation ID: aa1fc101-b074-42ef-8993-73eda65dca85
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw-np-upgrade-4v-<id>/cluster-npupgrade-4-21-<id> (service: virtualmachine): context deadline exceeded Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw-np-upgrade-4v-<id>/cluster-npupgrade-4-21-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x36e45b0b4740>:
failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-4vk5vd" in resourcegroup="rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c47db55b-db55-4983-b755-ff96a644195b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c47db55b-db55-4983-b755-ff96a644195b",
"name": "c47db55b-db55-4983-b755-ff96a644195b",
"status": "Failed",
"startTime": "2026-11-04T21:03:52.316353946Z",
"endTime": "2026-11-04T21:22:52.598672749Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-21-9twkn-c2mzj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npupgrade-4-21-9twkn-mkt55: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-21-9twkn-c2mzj: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw-np-upgrade-4v-30a0a03c/cluster-npupgrade-4-21-9twkn-c2mzj (service: virtualmachine): context deadline exceeded\nMachine cluster-npupgrade-4-21-9twkn-mkt55: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw-np-upgrade-4v-30a0a03c/cluster-npupgrade-4-21-9twkn-mkt55 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c47db55b-db55-4983-b755-ff96a644195b
correlation ID: 11ceb785-2646-4711-a8a9-cfe2ea00b130
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x36e45b0b4740>:
failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-4vk5vd" in resourcegroup="rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c47db55b-db55-4983-b755-ff96a644195b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c47db55b-db55-4983-b755-ff96a644195b",
"name": "c47db55b-db55-4983-b755-ff96a644195b",
"status": "Failed",
"startTime": "2026-11-04T21:03:52.316353946Z",
"endTime": "2026-11-04T21:22:52.598672749Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-21-9twkn-c2mzj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npupgrade-4-21-9twkn-mkt55: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-21-9twkn-c2mzj: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw-np-upgrade-4v-30a0a03c/cluster-npupgrade-4-21-9twkn-c2mzj (service: virtualmachine): context deadline exceeded\nMachine cluster-npupgrade-4-21-9twkn-mkt55: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-4vk5vd-n7jrdhgg65nw-np-upgrade-4v-30a0a03c/cluster-npupgrade-4-21-9twkn-mkt55 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c47db55b-db55-4983-b755-ff96a644195b
correlation ID: 11ceb785-2646-4711-a8a9-cfe2ea00b130
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-jkl87w-f2gtphdhv2ht-np-upgrade-jk-309d64f2/cluster-npupgrade-4-20-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x1bbd1f51bb60>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jkl87w" in resourcegroup="rg-np-version-upgrade-jkl87w-f2gtphdhv2ht" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/458c3303-89cc-403e-ad96-f7867cb5d073
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/458c3303-89cc-403e-ad96-f7867cb5d073",
"name": "458c3303-89cc-403e-ad96-f7867cb5d073",
"status": "Failed",
"startTime": "2026-11-04T21:03:10.292243588Z",
"endTime": "2026-11-04T21:22:12.567902678Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-ckb2r-ftsq6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-ckb2r-ftsq6: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-jkl87w-f2gtphdhv2ht-np-upgrade-jk-309d64f2/cluster-npupgrade-4-20-ckb2r-ftsq6 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/458c3303-89cc-403e-ad96-f7867cb5d073
correlation ID: cde161c0-fa12-4636-b309-e611bf5175ed
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x1bbd1f51bb60>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jkl87w" in resourcegroup="rg-np-version-upgrade-jkl87w-f2gtphdhv2ht" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/458c3303-89cc-403e-ad96-f7867cb5d073
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/458c3303-89cc-403e-ad96-f7867cb5d073",
"name": "458c3303-89cc-403e-ad96-f7867cb5d073",
"status": "Failed",
"startTime": "2026-11-04T21:03:10.292243588Z",
"endTime": "2026-11-04T21:22:12.567902678Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-ckb2r-ftsq6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-ckb2r-ftsq6: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-version-upgrade-jkl87w-f2gtphdhv2ht-np-upgrade-jk-309d64f2/cluster-npupgrade-4-20-ckb2r-ftsq6 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/458c3303-89cc-403e-ad96-f7867cb5d073
correlation ID: cde161c0-fa12-4636-b309-e611bf5175ed
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: AzureMachine status.initialization.provisioned is false; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
<*fmt.wrapError | 0x113f96974720>:
failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-trm2hk" in resourcegroup="rg-np-dg-minor-trm2hk-khnl2gf7fk9d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b62e489e-ee19-47e6-b1f7-80b932098058
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b62e489e-ee19-47e6-b1f7-80b932098058",
"name": "b62e489e-ee19-47e6-b1f7-80b932098058",
"status": "Failed",
"startTime": "2026-11-04T21:04:58.592948653Z",
"endTime": "2026-11-04T21:24:02.599850832Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npdg-422-420-7sn4x-5954f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npdg-422-420-7sn4x-5954f: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b62e489e-ee19-47e6-b1f7-80b932098058
correlation ID: 28d98513-bf97-4560-aecf-09cd4532a289
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
<*fmt.wrapError | 0x113f96974720>:
failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-trm2hk" in resourcegroup="rg-np-dg-minor-trm2hk-khnl2gf7fk9d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b62e489e-ee19-47e6-b1f7-80b932098058
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b62e489e-ee19-47e6-b1f7-80b932098058",
"name": "b62e489e-ee19-47e6-b1f7-80b932098058",
"status": "Failed",
"startTime": "2026-11-04T21:04:58.592948653Z",
"endTime": "2026-11-04T21:24:02.599850832Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-npdg-422-420-7sn4x-5954f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-npdg-422-420-7sn4x-5954f: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b62e489e-ee19-47e6-b1f7-80b932098058
correlation ID: 28d98513-bf97-4560-aecf-09cd4532a289
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-gqv2s9" for cluster "cluster-candidate-5-1-gqv2s9"
Unexpected error:
<*fmt.wrapError | 0x20e5f9d32d40>:
failed to create NodePool np-gqv2s9: failed waiting for nodepool="np-gqv2s9" for cluster "cluster-candidate-5-1-gqv2s9" in resourcegroup="rg-candidate-5-1-rsvdw6bx2nkn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/7e4ca83e-776e-48a0-804c-eb05ec7e3f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/7e4ca83e-776e-48a0-804c-eb05ec7e3f26",
"name": "7e4ca83e-776e-48a0-804c-eb05ec7e3f26",
"status": "Failed",
"startTime": "2026-11-04T17:01:42.913398283Z",
"endTime": "2026-11-04T17:20:51.877219716Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-gqv2s9-4zmnz-xd6j2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-rsvdw6bx2nkn-candidate-gqv2s9--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-gqv2s9-4zmnz-xd6j2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-gqv2s9-4zmnz-xd6j2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-rsvdw6bx2nkn-candidate-gqv2s9--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-gqv2s9-4zmnz-xd6j2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/7e4ca83e-776e-48a0-804c-eb05ec7e3f26
correlation ID: a6318e1d-7b3a-4bec-a769-233d819afa2e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-gqv2s9" for cluster "cluster-candidate-5-1-gqv2s9"
Unexpected error:
<*fmt.wrapError | 0x20e5f9d32d40>:
failed to create NodePool np-gqv2s9: failed waiting for nodepool="np-gqv2s9" for cluster "cluster-candidate-5-1-gqv2s9" in resourcegroup="rg-candidate-5-1-rsvdw6bx2nkn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/7e4ca83e-776e-48a0-804c-eb05ec7e3f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/7e4ca83e-776e-48a0-804c-eb05ec7e3f26",
"name": "7e4ca83e-776e-48a0-804c-eb05ec7e3f26",
"status": "Failed",
"startTime": "2026-11-04T17:01:42.913398283Z",
"endTime": "2026-11-04T17:20:51.877219716Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-gqv2s9-4zmnz-xd6j2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-rsvdw6bx2nkn-candidate-gqv2s9--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-gqv2s9-4zmnz-xd6j2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-gqv2s9-4zmnz-xd6j2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-rsvdw6bx2nkn-candidate-gqv2s9--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-gqv2s9-4zmnz-xd6j2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/7e4ca83e-776e-48a0-804c-eb05ec7e3f26
correlation ID: a6318e1d-7b3a-4bec-a769-233d819afa2e
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-lmvmnc-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-lmvmnc-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-lmvmnc" for cluster "cluster-candidate-4-20-lmvmnc"
Unexpected error:
<*fmt.wrapError | 0x233caeba7260>:
failed to create NodePool np-lmvmnc: failed waiting for nodepool="np-lmvmnc" for cluster "cluster-candidate-4-20-lmvmnc" in resourcegroup="rg-candidate-4-20-sslkgr7jnpq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32b67621-8792-4bef-ae3e-60b8bc2b258c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32b67621-8792-4bef-ae3e-60b8bc2b258c",
"name": "32b67621-8792-4bef-ae3e-60b8bc2b258c",
"status": "Failed",
"startTime": "2026-11-04T20:24:42.326155661Z",
"endTime": "2026-11-04T20:43:43.200393801Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-lmvmnc-h22k9-7qbsb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-sslkgr7jnpq5-candidate-lmvmnc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-lmvmnc-h22k9-7qbsb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-lmvmnc-h22k9-7qbsb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-sslkgr7jnpq5-candidate-lmvmnc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-lmvmnc-h22k9-7qbsb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32b67621-8792-4bef-ae3e-60b8bc2b258c
correlation ID: 6ef67b4a-bf1e-4b07-9ed9-227bcc49369e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-lmvmnc" for cluster "cluster-candidate-4-20-lmvmnc"
Unexpected error:
<*fmt.wrapError | 0x233caeba7260>:
failed to create NodePool np-lmvmnc: failed waiting for nodepool="np-lmvmnc" for cluster "cluster-candidate-4-20-lmvmnc" in resourcegroup="rg-candidate-4-20-sslkgr7jnpq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32b67621-8792-4bef-ae3e-60b8bc2b258c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32b67621-8792-4bef-ae3e-60b8bc2b258c",
"name": "32b67621-8792-4bef-ae3e-60b8bc2b258c",
"status": "Failed",
"startTime": "2026-11-04T20:24:42.326155661Z",
"endTime": "2026-11-04T20:43:43.200393801Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-lmvmnc-h22k9-7qbsb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-sslkgr7jnpq5-candidate-lmvmnc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-lmvmnc-h22k9-7qbsb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-lmvmnc-h22k9-7qbsb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-sslkgr7jnpq5-candidate-lmvmnc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-lmvmnc-h22k9-7qbsb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32b67621-8792-4bef-ae3e-60b8bc2b258c
correlation ID: 6ef67b4a-bf1e-4b07-9ed9-227bcc49369e
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-wgmwth-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-wgmwth-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-wgmwth" for cluster "cluster-candidate-5-0-wgmwth"
Unexpected error:
<*fmt.wrapError | 0x180da9fa3740>:
failed to create NodePool np-wgmwth: failed waiting for nodepool="np-wgmwth" for cluster "cluster-candidate-5-0-wgmwth" in resourcegroup="rg-candidate-5-0-45259f2vftxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f49a37db-8fd1-4041-bad1-ab335641c6b0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f49a37db-8fd1-4041-bad1-ab335641c6b0",
"name": "f49a37db-8fd1-4041-bad1-ab335641c6b0",
"status": "Failed",
"startTime": "2026-11-04T20:28:26.524569768Z",
"endTime": "2026-11-04T20:47:33.242857363Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-wgmwth-4c5bk-29m4l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-45259f2vftxm-candidate-wgmwth--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-wgmwth-4c5bk-29m4l to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-wgmwth-4c5bk-29m4l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-45259f2vftxm-candidate-wgmwth--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-wgmwth-4c5bk-29m4l to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f49a37db-8fd1-4041-bad1-ab335641c6b0
correlation ID: 67dc3a1b-7d46-460e-9bb4-a791961e28a5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-wgmwth" for cluster "cluster-candidate-5-0-wgmwth"
Unexpected error:
<*fmt.wrapError | 0x180da9fa3740>:
failed to create NodePool np-wgmwth: failed waiting for nodepool="np-wgmwth" for cluster "cluster-candidate-5-0-wgmwth" in resourcegroup="rg-candidate-5-0-45259f2vftxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f49a37db-8fd1-4041-bad1-ab335641c6b0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f49a37db-8fd1-4041-bad1-ab335641c6b0",
"name": "f49a37db-8fd1-4041-bad1-ab335641c6b0",
"status": "Failed",
"startTime": "2026-11-04T20:28:26.524569768Z",
"endTime": "2026-11-04T20:47:33.242857363Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-wgmwth-4c5bk-29m4l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-45259f2vftxm-candidate-wgmwth--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-wgmwth-4c5bk-29m4l to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-wgmwth-4c5bk-29m4l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-45259f2vftxm-candidate-wgmwth--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-wgmwth-4c5bk-29m4l to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f49a37db-8fd1-4041-bad1-ab335641c6b0
correlation ID: 67dc3a1b-7d46-460e-9bb4-a791961e28a5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; hypershift NodePool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; hypershift NodePool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-labels-taints-z8n8hhzcmfrh--managed/cluster-np-1-mglnf-<id> (service: virtualmachine): context deadline exceeded; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:228]: failed to update node pool np-1 with new label and scale up
Unexpected error:
<*fmt.wrapError | 0x10912aa6e260>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-z8n8hhzcmfrh" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e43ec58f-c807-49fd-8e6a-6d5d59d68f51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e43ec58f-c807-49fd-8e6a-6d5d59d68f51",
"name": "e43ec58f-c807-49fd-8e6a-6d5d59d68f51",
"status": "Failed",
"startTime": "2026-11-04T21:12:55.024814661Z",
"endTime": "2026-11-04T21:31:55.433481865Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is 3, want 4; hypershift NodePool condition AllNodesHealthy is False: 1 of 4 machines are not healthy\nMachine cluster-np-1-mglnf-l5pg4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; hypershift NodePool condition AllMachinesReady is False: 1 of 4 machines are not ready\nMachine cluster-np-1-mglnf-l5pg4: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-labels-taints-z8n8hhzcmfrh--managed/cluster-np-1-mglnf-l5pg4 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e43ec58f-c807-49fd-8e6a-6d5d59d68f51
correlation ID: 72f17fbd-0982-432c-b491-9d3d6930bc4e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:228]: failed to update node pool np-1 with new label and scale up
Unexpected error:
<*fmt.wrapError | 0x10912aa6e260>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-z8n8hhzcmfrh" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e43ec58f-c807-49fd-8e6a-6d5d59d68f51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e43ec58f-c807-49fd-8e6a-6d5d59d68f51",
"name": "e43ec58f-c807-49fd-8e6a-6d5d59d68f51",
"status": "Failed",
"startTime": "2026-11-04T21:12:55.024814661Z",
"endTime": "2026-11-04T21:31:55.433481865Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool update did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is updating; [hypershiftNodePool] hypershift NodePool status replicas is 3, want 4; hypershift NodePool condition AllNodesHealthy is False: 1 of 4 machines are not healthy\nMachine cluster-np-1-mglnf-l5pg4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; hypershift NodePool condition AllMachinesReady is False: 1 of 4 machines are not ready\nMachine cluster-np-1-mglnf-l5pg4: Failed: virtualmachine failed to create or update. err: failed to create or update resource rg-np-labels-taints-z8n8hhzcmfrh--managed/cluster-np-1-mglnf-l5pg4 (service: virtualmachine): context deadline exceeded\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e43ec58f-c807-49fd-8e6a-6d5d59d68f51
correlation ID: 72f17fbd-0982-432c-b491-9d3d6930bc4e
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpe...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 1 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xca03c226ff0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 43ef136d-3363-4bff-be35-8603a255e4de
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xca03c226ff0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 43ef136d-3363-4bff-be35-8603a255e4de
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: StorageAccountIsNotProvisioned; provider Microsoft.Storagefull failure pattern: ERROR CODE: StorageAccountIsNotProvisioned; provider Microsoft.Storage Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-4-22-wv7l8b"
Unexpected error:
<*errors.joinError | 0x1d915a6eae10>:
VerifyCSIPersistentVolume(file.csi.azure.com) failed: VerifyCSIPersistentVolume(file.csi.azure.com) timed out after 10m0.003s: PVC e2e-csi-file-28w79/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
[storageclass] name=azurefile-csi provisioner=file.csi.azure.com
[pvc] e2e-csi-file-28w79/e2e-csi-pvc phase=Pending volumeName= storageClass=azurefile-csi
[pod] e2e-csi-file-28w79/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodScheduled status=False reason=Unschedulable message=0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=10m0s)
[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=9m53s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=4m16s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-file-28w79/e2e-csi-pvc" (reason=Provisioning, count=9, age=43s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=1, age=5m0s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = Internal desc = failed to ensure storage account: could not get storage key for storage account feeda6dfe1a31466189e73a: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-mv5mkd96jrfl-candidate-wv7l8b--managed/providers/Microsoft.Storage/storageAccounts/feeda6dfe1a31466189e73a/listKeys
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: StorageAccountIsNotProvisioned
--------------------------------------------------------------------------------
{
"error": {
"code": "StorageAccountIsNotProvisioned",
"message": "The storage account provisioning state must be 'Succeeded' before executing the operation."
}
}
--------------------------------------------------------------------------------
(reason=ProvisioningFailed, count=8, age=42s)
{
errs: [
<*fmt.wrapError | 0x1d915a16ca80>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) failed: VerifyCSIPersistentVolume(file.csi.azure.com) timed out after 10m0.003s: PVC e2e-csi-file-28w79/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=azurefile-csi provisioner=file.csi.azure.com\n[pvc] e2e-csi-file-28w79/e2e-csi-pvc phase=Pending volumeName= storageClass=azurefile-csi\n[pod] e2e-csi-file-28w79/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=Unschedulable message=0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=10m0s)\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=9m53s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=4m16s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-file-28w79/e2e-csi-pvc\" (reason=Provisioning, count=9, age=43s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=1, age=5m0s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = Internal desc = failed to ensure storage account: could not get storage key for storage account feeda6dfe1a31466189e73a: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-mv5mkd96jrfl-candidate-wv7l8b--managed/providers/Microsoft.Storage/storageAccounts/feeda6dfe1a31466189e73a/listKeys\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: StorageAccountIsNotProvisioned\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"StorageAccountIsNotProvisioned\",\n \"message\": \"The storage account provisioning state must be 'Succeeded' before executing the operation.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=ProvisioningFailed, count=8, age=42s)",
err: <*fmt.wrapError | 0x1d915a16c7a0>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) timed out after 10m0.003s: PVC e2e-csi-file-28w79/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=azurefile-csi provisioner=file.csi.azure.com\n[pvc] e2e-csi-file-28w79/e2e-csi-pvc phase=Pending volumeName= storageClass=azurefile-csi\n[pod] e2e-csi-file-28w79/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=Unschedulable message=0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=10m0s)\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=9m53s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=4m16s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-file-28w79/e2e-csi-pvc\" (reason=Provisioning, count=9, age=43s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=1, age=5m0s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = Internal desc = failed to ensure storage account: could not get storage key for storage account feeda6dfe1a31466189e73a: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-mv5mkd96jrfl-candidate-wv7l8b--managed/providers/Microsoft.Storage/storageAccounts/feeda6dfe1a31466189e73a/listKeys\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: StorageAccountIsNotProvisioned\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"StorageAccountIsNotProvisioned\",\n \"message\": \"The storage account provisioning state must be 'Succeeded' before executing the operation.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=ProvisioningFailed, count=8, age=42s)",
err: <*errors.errorString | 0x1d9159fca870>{
s: "PVC e2e-csi-file-28w79/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-4-22-wv7l8b"
Unexpected error:
<*errors.joinError | 0x1d915a6eae10>:
VerifyCSIPersistentVolume(file.csi.azure.com) failed: VerifyCSIPersistentVolume(file.csi.azure.com) timed out after 10m0.003s: PVC e2e-csi-file-28w79/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
[storageclass] name=azurefile-csi provisioner=file.csi.azure.com
[pvc] e2e-csi-file-28w79/e2e-csi-pvc phase=Pending volumeName= storageClass=azurefile-csi
[pod] e2e-csi-file-28w79/e2e-csi-pod phase=Pending reason= message=
[pod] condition type=PodScheduled status=False reason=Unschedulable message=0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=10m0s)
[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=9m53s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=4m16s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-file-28w79/e2e-csi-pvc" (reason=Provisioning, count=9, age=43s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=1, age=5m0s)
[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = Internal desc = failed to ensure storage account: could not get storage key for storage account feeda6dfe1a31466189e73a: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-mv5mkd96jrfl-candidate-wv7l8b--managed/providers/Microsoft.Storage/storageAccounts/feeda6dfe1a31466189e73a/listKeys
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: StorageAccountIsNotProvisioned
--------------------------------------------------------------------------------
{
"error": {
"code": "StorageAccountIsNotProvisioned",
"message": "The storage account provisioning state must be 'Succeeded' before executing the operation."
}
}
--------------------------------------------------------------------------------
(reason=ProvisioningFailed, count=8, age=42s)
{
errs: [
<*fmt.wrapError | 0x1d915a16ca80>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) failed: VerifyCSIPersistentVolume(file.csi.azure.com) timed out after 10m0.003s: PVC e2e-csi-file-28w79/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=azurefile-csi provisioner=file.csi.azure.com\n[pvc] e2e-csi-file-28w79/e2e-csi-pvc phase=Pending volumeName= storageClass=azurefile-csi\n[pod] e2e-csi-file-28w79/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=Unschedulable message=0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=10m0s)\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=9m53s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=4m16s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-file-28w79/e2e-csi-pvc\" (reason=Provisioning, count=9, age=43s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=1, age=5m0s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = Internal desc = failed to ensure storage account: could not get storage key for storage account feeda6dfe1a31466189e73a: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-mv5mkd96jrfl-candidate-wv7l8b--managed/providers/Microsoft.Storage/storageAccounts/feeda6dfe1a31466189e73a/listKeys\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: StorageAccountIsNotProvisioned\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"StorageAccountIsNotProvisioned\",\n \"message\": \"The storage account provisioning state must be 'Succeeded' before executing the operation.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=ProvisioningFailed, count=8, age=42s)",
err: <*fmt.wrapError | 0x1d915a16c7a0>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) timed out after 10m0.003s: PVC e2e-csi-file-28w79/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=azurefile-csi provisioner=file.csi.azure.com\n[pvc] e2e-csi-file-28w79/e2e-csi-pvc phase=Pending volumeName= storageClass=azurefile-csi\n[pod] e2e-csi-file-28w79/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=Unschedulable message=0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=10m0s)\n[events] e2e-csi-file-28w79 Pod/e2e-csi-pod: 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found (reason=FailedScheduling, count=0, age=9m53s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=4m16s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-file-28w79/e2e-csi-pvc\" (reason=Provisioning, count=9, age=43s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=1, age=5m0s)\n[events] e2e-csi-file-28w79 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = Internal desc = failed to ensure storage account: could not get storage key for storage account feeda6dfe1a31466189e73a: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-mv5mkd96jrfl-candidate-wv7l8b--managed/providers/Microsoft.Storage/storageAccounts/feeda6dfe1a31466189e73a/listKeys\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: StorageAccountIsNotProvisioned\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"StorageAccountIsNotProvisioned\",\n \"message\": \"The storage account provisioning state must be 'Succeeded' before executing the operation.\"\n }\n}\n--------------------------------------------------------------------------------\n (reason=ProvisioningFailed, count=8, age=42s)",
err: <*errors.errorString | 0x1d9159fca870>{
s: "PVC e2e-csi-file-28w79/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-<id> namespace are ready: test pods not yet ready: [host-to-b-multi-nod...full failure pattern: VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-<id> namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-<id> (container host-to-b-multi-node-clusterip-container waiting Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0x30333fd91cb0>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-jp7x5p namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]
{
errs: [
<*fmt.wrapError | 0x303340854220>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-jp7x5p namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]",
err: <*fmt.wrapError | 0x30333f49fc00>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-jp7x5p namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]",
err: <*errors.errorString | 0x303342496830>{
s: "test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]",
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0x30333fd91cb0>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-jp7x5p namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]
{
errs: [
<*fmt.wrapError | 0x303340854220>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-jp7x5p namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]",
err: <*fmt.wrapError | 0x30333f49fc00>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-jp7x5p namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]",
err: <*errors.errorString | 0x303342496830>{
s: "test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-njgf4 (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-mq529 (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-7674f (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-cd5x2 (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-nodeport-6c7bfb67c8-jm4ck (container pod-to-b-multi-node-nodeport-container waiting: CrashLoopBackOff)]",
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
creating Certificate: Post "<url>": read tcp <ip>full failure pattern: creating Certificate: Post "<url>": read tcp <ip> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 1 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:448]: creating Certificate: Post "https://api.cluster.skl6b.eastus2euap.aroapp-hcp.io:443/apis/cert-manager.io/v1/namespaces/openshift-ingress/certificates": read tcp 10.129.49.125:51428->72.147.62.64:443: read: connection reset by peer fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:448]: creating Certificate: Post "https://api.cluster.skl6b.eastus2euap.aroapp-hcp.io:443/apis/cert-manager.io/v1/namespaces/openshift-ingress/certificates": read tcp 10.129.49.125:51428->72.147.62.64:443: read: connection reset by peer Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x78ccfac49c0>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gbp7qkhtz9q9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc778bba-a3b1-42de-9409-948721112d66
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc778bba-a3b1-42de-9409-948721112d66",
"name": "fc778bba-a3b1-42de-9409-948721112d66",
"status": "Failed",
"startTime": "2026-11-04T20:57:51.407498968Z",
"endTime": "2026-11-04T21:16:52.545188415Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-delete-main-q8k6n-t28c6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-delete-main-q8k6n-t28c6: Failed: virtualmachine failed to create or update. err: failed to reapply VM nodepool-delete-gbp7qkhtz9q9--managed/cluster-np-delete-main-q8k6n-t28c6: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-gbp7qkhtz9q9--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-delete-main-q8k6n-t28c6/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'a2599f46-4d42-4647-a963-b39bfd3d385a' with object id '3b910b41-5d16-46ce-9221-b7aee45ed7d6' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resou... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc778bba-a3b1-42de-9409-948721112d66
correlation ID: 12a0a4bf-fa64-4321-b09b-a1cc047a78c7
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x78ccfac49c0>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gbp7qkhtz9q9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc778bba-a3b1-42de-9409-948721112d66
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc778bba-a3b1-42de-9409-948721112d66",
"name": "fc778bba-a3b1-42de-9409-948721112d66",
"status": "Failed",
"startTime": "2026-11-04T20:57:51.407498968Z",
"endTime": "2026-11-04T21:16:52.545188415Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-delete-main-q8k6n-t28c6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-delete-main-q8k6n-t28c6: Failed: virtualmachine failed to create or update. err: failed to reapply VM nodepool-delete-gbp7qkhtz9q9--managed/cluster-np-delete-main-q8k6n-t28c6: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-gbp7qkhtz9q9--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-delete-main-q8k6n-t28c6/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'a2599f46-4d42-4647-a963-b39bfd3d385a' with object id '3b910b41-5d16-46ce-9221-b7aee45ed7d6' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resou... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fc778bba-a3b1-42de-9409-948721112d66
correlation ID: 12a0a4bf-fa64-4321-b09b-a1cc047a78c7
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadlinefull failure pattern: failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x3c7718736630>:
context deadline exceeded
failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
{
errs: [
<context.deadlineExceededError>{},
<*fmt.wrapError | 0x3c7716857520>{
msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x3c77168574e0>{
msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*errors.errorString | 0x3c7718606890>{
s: "rate: Wait(n=1) would exceed context deadline",
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x3c7718736630>:
context deadline exceeded
failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
{
errs: [
<context.deadlineExceededError>{},
<*fmt.wrapError | 0x3c7716857520>{
msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x3c77168574e0>{
msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*errors.errorString | 0x3c7718606890>{
s: "rate: Wait(n=1) would exceed context deadline",
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceededfull failure pattern: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
<*fmt.wrapError | 0x207eadb0860>:
failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-964hb55zckvs" to finish creating: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-964hb55zckvs\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
<*fmt.wrapError | 0x207eadb0860>:
failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-964hb55zckvs" to finish creating: context deadline exceeded
{
msg: "failed waiting for nodepool=\"np-ver-4-20-32\" for cluster \"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-964hb55zckvs\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
storage account log verification failedfull failure pattern: storage account log verification failed Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.00%1 of 50 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 1 · Nov 5: 0 · Nov 6: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox5p6ghz and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-z58z7w5dkksd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
<*errors.errorString | 0x2ae9ebba32b0>:
storage account log verification failed
{
s: "storage account log verification failed",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox5p6ghz and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-z58z7w5dkksd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
<*errors.errorString | 0x2ae9ebba32b0>:
storage account log verification failed
{
s: "storage account log verification failed",
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||