Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 136 | 48.53% (off track) | -46.47pp | +3.53pp | 92.56% (112/121) | +2.41pp | 58.93% (66/112) | +4.83pp |
| INT | 95% - All E2E job runs | 36 | 80.56% (off track) | -14.44pp | +45.67pp | n/a | n/a | 100.00% (29/29) | +0.00pp |
| STG | 95% - All E2E job runs | 30 | 33.33% (off track) | -61.67pp | -1.28pp | n/a | n/a | 76.92% (10/13) | +31.92pp |
| PROD | 95% - All E2E job runs | 52 | 59.62% (off track) | -35.38pp | -0.88pp | n/a | n/a | 72.09% (31/43) | +5.88pp |
Environment: DEV
E2E Jobs
313
Success Rate
37.38%
Provision success
Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.90.53% (239/264)
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.48.95% (117/239)
E2E Jobs (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the number of runs in that DEV-only goal basis.136
Success Rate (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the overall success rate for that DEV-only goal basis.48.53%
Provision success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.92.56% (112/121)
E2E success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.58.93% (66/112)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
Successful runs (post-good + batches)
Provision failures
E2E failures
Other failures
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 13 | 4.15%13 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 77 · Nov 4: 0 | INT, PROD, STG | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1a0ca8223b30>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1a0ca8223b30>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3c8924320290>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3c8924320290>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x36a7d1d57b70>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x36a7d1d57b70>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (13) | |||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | unknown | 13 | 4.15%13 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsOct 28: 0 · Oct 29: 3 · Oct 30: 4 · Oct 31: 0 · Nov 1: 1 · Nov 2: 2 · Nov 3: 3 · Nov 4: 0 | INT, PROD, STG | |||||||||||||||||||||||||||||||
Full failure examples (3){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-03T19:00:22Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-03T15:30:08Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-03T09:40:55Z"}Contributing tests (1)
Affected runs (13) | |||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | provision | 8 | 2.56%8 of 313 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 28: 3 · Oct 29: 1 · Oct 30: 3 · Oct 31: 0 · Nov 1: 0 · Nov 2: 3 · Nov 3: 1 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-03T19:59:54.828Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-03T19:59:56.107Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4?api-version=2021-04-01"
time=2026-11-03T19:59:58.238Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4?api-version=2021-04-01"
time=2026-11-03T20:00:02.411Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4?api-version=2021-04-01"
time=2026-11-03T20:00:11.512Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4?api-version=2021-04-01"
time=2026-11-03T20:00:28.416Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4?api-version=2021-04-01"
time=2026-11-03T20:01:02.612Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4?api-version=2021-04-01"
time=2026-11-03T20:01:02.612Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-03T20:01:03.886Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3461632-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4
time=2026-11-03T20:03:34.577Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/20349c2a3d0560255c5ce9d6451596f7118b661b7cf706423e623a45b2f606b4/operationStatuses/08584078708222295736\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: c46a985c-9e95-4143-8608-4f9fb0e41766\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/cluster-krwy2abqgscze\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: c46a985c-9e95-4143-8608-4f9fb0e41766\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j3461632-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j3461632-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-03T20:05:34.582Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-03T20:05:35.739Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd?api-version=2021-04-01"
time=2026-11-03T20:05:38.013Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd?api-version=2021-04-01"
time=2026-11-03T20:05:42.407Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd?api-version=2021-04-01"
time=2026-11-03T20:05:50.548Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd?api-version=2021-04-01"
time=2026-11-03T20:06:06.760Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd?api-version=2021-04-01"
time=2026-11-03T20:06:39.221Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd?api-version=2021-04-01"
time=2026-11-03T20:06:39.221Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-03T20:06:41.106Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3461632-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd
time=2026-11-03T20:08:11.415Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3461632-svc/providers/Microsoft.Resources/deployments/795bad99c8b5f5e71af3b0ccc5126610f71842169a6e2408c874a643064573dd/operationStatuses/08584078704854684525\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: d019f8af-c085-465b-aa0a-9bbc88aea379\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-02T20:05:19.707Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-02T20:05:21.093Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1?api-version=2021-04-01"
time=2026-11-02T20:05:23.277Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1?api-version=2021-04-01"
time=2026-11-02T20:05:27.602Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1?api-version=2021-04-01"
time=2026-11-02T20:05:36.142Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1?api-version=2021-04-01"
time=2026-11-02T20:05:53.029Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1?api-version=2021-04-01"
time=2026-11-02T20:06:26.944Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1?api-version=2021-04-01"
time=2026-11-02T20:06:26.944Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-02T20:06:27.893Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4235136-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1
time=2026-11-02T20:09:28.395Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/96c58b208347c2a558304db1970440e85266d4149c499487684bef18270e07b1/operationStatuses/08584079568980380477\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 7835ece5-7cdc-42eb-9bbb-5aca6a408ced\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/cluster-cnuj6kkpv6amy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 7835ece5-7cdc-42eb-9bbb-5aca6a408ced\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j4235136-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j4235136-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-02T20:11:28.398Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-02T20:11:29.150Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77?api-version=2021-04-01"
time=2026-11-02T20:11:31.365Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77?api-version=2021-04-01"
time=2026-11-02T20:11:35.732Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77?api-version=2021-04-01"
time=2026-11-02T20:11:44.008Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77?api-version=2021-04-01"
time=2026-11-02T20:12:01.470Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77?api-version=2021-04-01"
time=2026-11-02T20:12:35.715Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77?api-version=2021-04-01"
time=2026-11-02T20:12:35.715Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-02T20:12:36.695Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4235136-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77
time=2026-11-02T20:14:07.617Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4235136-svc/providers/Microsoft.Resources/deployments/82dc38dc384f8fa2900dc598046d446d55ae6f50e4bfe561c36c0b975e1bfb77/operationStatuses/08584079565292751324\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8e66f419-7f34-4d24-9681-31c973a5600b\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-02T19:30:01.260Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-02T19:30:03.242Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052?api-version=2021-04-01"
time=2026-11-02T19:30:05.385Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052?api-version=2021-04-01"
time=2026-11-02T19:30:09.641Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052?api-version=2021-04-01"
time=2026-11-02T19:30:17.785Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052?api-version=2021-04-01"
time=2026-11-02T19:30:36.232Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052?api-version=2021-04-01"
time=2026-11-02T19:31:10.996Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052?api-version=2021-04-01"
time=2026-11-02T19:31:10.996Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-02T19:31:12.232Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3674624-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052
time=2026-11-02T19:33:42.748Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/8e88888d47f7272f51eab2835a0cb3e79f331e5ffbd0c6b4b7e68169001e6052/operationStatuses/08584079590138245686\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 115f0cc4-01fa-4e27-b23d-74329aa8780f\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/cluster-bducmbmjc445y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 115f0cc4-01fa-4e27-b23d-74329aa8780f\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j3674624-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j3674624-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-02T19:35:42.749Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-02T19:35:43.585Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59?api-version=2021-04-01"
time=2026-11-02T19:35:45.703Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59?api-version=2021-04-01"
time=2026-11-02T19:35:49.838Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59?api-version=2021-04-01"
time=2026-11-02T19:35:58.643Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59?api-version=2021-04-01"
time=2026-11-02T19:36:15.439Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59?api-version=2021-04-01"
time=2026-11-02T19:36:50.457Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59?api-version=2021-04-01"
time=2026-11-02T19:36:50.457Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-02T19:36:52.043Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3674624-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59
time=2026-11-02T19:38:23.799Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3674624-svc/providers/Microsoft.Resources/deployments/4765ded6a8c946ea23c83ae807681f5e3d9b8e9018d3d28ef6d6f52d7313fe59/operationStatuses/08584079586742464323\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b317ae49-f5da-450f-b60b-3ea94888526e\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| |||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 7 | 2.24%7 of 313 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 28: 3 · Oct 29: 2 · Oct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 3 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-02T23:57:05Z Ended: 2026-11-03T00:18:03Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6567680-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster", pod="router-645764895d-lbncd", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster/router-645764895d-lbncd has been in a non-ready state for longer than 5 minutes. alert fired 38 time(s) (19 unknown, 19 known) Firing 1: State: Resolved Started: 2026-11-02T17:11:31Z Ended: 2026-11-02T17:20:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr77sj3qpl4hs9fsbnu4a6rdr4olq38-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr77sj3qpl4hs9fsbnu4a6rdr4olq38-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-02T17:11:31Z Ended: 2026-11-02T17:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr785bj7ssg4tki1t502kjiu4kf7t28-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr785bj7ssg4tki1t502kjiu4kf7t28-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-02T17:11:31Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78dop5voo1j2vbl02fd0c9sbm5ovn-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78dop5voo1j2vbl02fd0c9sbm5ovn-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr77ughilvolgkvokq8n5gmveu7m7bj-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr77ughilvolgkvokq8n5gmveu7m7bj-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78j01jd8mmt682pl0kmgqao3ronrl-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78j01jd8mmt682pl0kmgqao3ronrl-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr785lf5nsal1cgst1n6q27kb7ka1te-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr785lf5nsal1cgst1n6q27kb7ka1te-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr781r5qjooeqf24frcai2l3r83c2nk-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr781r5qjooeqf24frcai2l3r83c2nk-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78el8p6a1upghr6q8ohbjuvjqimu5-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78el8p6a1upghr6q8ohbjuvjqimu5-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:22:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr77s49v0rkal88n5mjmmrk2cqnsqfl-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr77s49v0rkal88n5mjmmrk2cqnsqfl-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr7886ln9uob4cnaa2j966h20n1552l-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr7886ln9uob4cnaa2j966h20n1552l-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-11-02T17:12:29Z Ended: 2026-11-02T17:20:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78mjuia2omhmbms0gpkjk850tsvqk-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78mjuia2omhmbms0gpkjk850tsvqk-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-11-02T17:13:29Z Ended: 2026-11-02T17:22:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78km2hs0h9gn8680f4kev09f85d6i-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78km2hs0h9gn8680f4kev09f85d6i-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-11-02T17:13:29Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr781k9h0kn80cou563hbe6qjs68439-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr781k9h0kn80cou563hbe6qjs68439-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-11-02T17:13:29Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr782mn3cvrvsi6erfdd59jhfclg5v5-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr782mn3cvrvsi6erfdd59jhfclg5v5-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-11-02T17:13:30Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78fi8j3nf1vcianmk2di9b38gogj2-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78fi8j3nf1vcianmk2di9b38gogj2-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-11-02T17:13:31Z Ended: 2026-11-02T17:20:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78dno0k1j8dqs0gpiqvvl0ckhuas7-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78dno0k1j8dqs0gpiqvvl0ckhuas7-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-11-02T17:13:31Z Ended: 2026-11-02T17:22:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr780qeppk7140o0skssraepc0drcu6-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr780qeppk7140o0skssraepc0drcu6-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-11-02T17:14:29Z Ended: 2026-11-02T17:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr78svn7ehgj2m8viqa2qle3e43aemp-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr78svn7ehgj2m8viqa2qle3e43aemp-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-11-02T17:15:30Z Ended: 2026-11-02T17:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3919104-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr794f27e7abj1f4n1fqdcp6kjn5ra9-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr794f27e7abj1f4n1fqdcp6kjn5ra9-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-02T10:19:01Z Ended: 2026-11-02T10:25:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2102912-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tr0vbkfjp3g8khsfsfaqdud2ngt9bir-cluster", pod="openshift-apiserver-9765fd46f-n2c5v", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tr0vbkfjp3g8khsfsfaqdud2ngt9bir-cluster/openshift-apiserver-9765fd46f-n2c5v has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (7) | |||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | alert | 7 | 2.24%7 of 313 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 28: 4 · Oct 29: 5 · Oct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-30T08:51:48Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9924864-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.179:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9924864/providers/microsoft.monitor/accounts/services-j9924864", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-jrsn7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="781a0983-5a00-433e-9c6b-0cc6c8b86a10" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-jrsn7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-30T08:52:48Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9924864-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.179:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9924864/providers/microsoft.monitor/accounts/services-j9924864", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-jrsn7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="781a0983-5a00-433e-9c6b-0cc6c8b86a10" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-jrsn7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-30T08:53:46Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9924864-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9924864/providers/microsoft.monitor/accounts/services-j9924864", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-l52wq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="63d2df6d-206a-4b4a-a6bf-deeab61bc9c4" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-l52wq (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-30T08:56:49Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9924864-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9924864/providers/microsoft.monitor/accounts/services-j9924864", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-l52wq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="63d2df6d-206a-4b4a-a6bf-deeab61bc9c4" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-l52wq (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-30T03:22:55Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j4747904-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4747904/providers/microsoft.monitor/accounts/services-j4747904", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-fdrhk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="19563b05-15fd-47c1-870b-b3970808eceb" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-fdrhk (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-30T03:22:55Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j4747904-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.96:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4747904/providers/microsoft.monitor/accounts/services-j4747904", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-qdc52", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="93fd4b63-6eb2-48ac-b3df-6838507688fb" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-qdc52 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-30T03:25:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j4747904-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.96:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4747904/providers/microsoft.monitor/accounts/services-j4747904", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-qdc52", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="93fd4b63-6eb2-48ac-b3df-6838507688fb" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-qdc52 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-30T03:25:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j4747904-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4747904/providers/microsoft.monitor/accounts/services-j4747904", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-fdrhk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="19563b05-15fd-47c1-870b-b3970808eceb" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-fdrhk (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-30T00:10:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7852800-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7852800/providers/microsoft.monitor/accounts/services-j7852800", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-zcb25", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fded3005-7c16-4438-9081-d6f8fa6947a6" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-zcb25 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-30T00:11:31Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7852800-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7852800/providers/microsoft.monitor/accounts/services-j7852800", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-mncdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="19872ef9-206e-4db9-b7f6-58c78be213dc" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-mncdw (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-30T00:13:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7852800-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7852800/providers/microsoft.monitor/accounts/services-j7852800", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-zcb25", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fded3005-7c16-4438-9081-d6f8fa6947a6" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-zcb25 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-30T00:13:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7852800-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7852800/providers/microsoft.monitor/accounts/services-j7852800", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-mncdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="19872ef9-206e-4db9-b7f6-58c78be213dc" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-mncdw (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (7) | |||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 6 | 1.92%6 of 313 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 28: 0 · Oct 29: 2 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 1 · Nov 4: 0 | PROD, STG | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xf7c689961e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-xbr2lbp7xpdp" to finish creating: GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"name": "91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"status": "Failed",
"startTime": "2026-11-03T18:00:37.406264058Z",
"endTime": "2026-11-03T18:19:39.541562946Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-wmfl5-9q262: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-wmfl5-txll2: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-xbr2lbp7xpdp--managed/cluster-np-1-wmfl5-txll2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-wmfl5-txll2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\nMachine cluster-np-1-wmfl5-9q262: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
correlation ID: 036f75f3-24b0-4f91-bc59-5746100a9a8a
client request ID: f269ac56-2518-4210-976b-61a6c750a6ce
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xf7c689961e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-xbr2lbp7xpdp" to finish creating: GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"name": "91ec8ee0-8d21-4d67-b056-6382b74b28b2",
"status": "Failed",
"startTime": "2026-11-03T18:00:37.406264058Z",
"endTime": "2026-11-03T18:19:39.541562946Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-wmfl5-9q262: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-wmfl5-txll2: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-xbr2lbp7xpdp--managed/cluster-np-1-wmfl5-txll2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-wmfl5-txll2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\nMachine cluster-np-1-wmfl5-9q262: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xbr2lbp7xpdp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-wmfl5-9q262 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7306240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91ec8ee0-8d21-4d67-b056-6382b74b28b2
correlation ID: 036f75f3-24b0-4f91-bc59-5746100a9a8a
client request ID: f269ac56-2518-4210-976b-61a6c750a6cefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1554a483a2c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fqqws26gj5wg" to finish creating: GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a",
"name": "99b5366a-e767-4906-ac36-4294137d7f8a",
"status": "Failed",
"startTime": "2026-11-02T18:21:28.76444558Z",
"endTime": "2026-11-02T18:40:30.672040856Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9jhrf-vj7z9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9jhrf-vj7z9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
correlation ID: d32d6140-7634-4f62-8bb9-b4e1bdf31356
client request ID: 313e9b91-f2fc-449a-b3b6-d509cbe4b657
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1554a483a2c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-fqqws26gj5wg" to finish creating: GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a",
"name": "99b5366a-e767-4906-ac36-4294137d7f8a",
"status": "Failed",
"startTime": "2026-11-02T18:21:28.76444558Z",
"endTime": "2026-11-02T18:40:30.672040856Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9jhrf-vj7z9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9jhrf-vj7z9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-fqqws26gj5wg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9jhrf-vj7z9 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5997952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99b5366a-e767-4906-ac36-4294137d7f8a
correlation ID: d32d6140-7634-4f62-8bb9-b4e1bdf31356
client request ID: 313e9b91-f2fc-449a-b3b6-d509cbe4b657fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x2e6ce0dd4880>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-45h85kz7nx75" to finish creating: GET https://rp.j2430848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3",
"name": "4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3",
"status": "Failed",
"startTime": "2026-11-02T05:51:50.21566157Z",
"endTime": "2026-11-02T06:10:56.082199009Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-5qcs2-l57tr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-45h85kz7nx75--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5qcs2-l57tr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-5qcs2-l57tr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-45h85kz7nx75--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5qcs2-l57tr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2430848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3
correlation ID: 3db1a137-1e98-4c4a-af29-550f7818979f
client request ID: 15091261-7ca5-4fc4-9cba-cb6f5b7430f3
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x2e6ce0dd4880>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-45h85kz7nx75" to finish creating: GET https://rp.j2430848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3",
"name": "4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3",
"status": "Failed",
"startTime": "2026-11-02T05:51:50.21566157Z",
"endTime": "2026-11-02T06:10:56.082199009Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-5qcs2-l57tr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-45h85kz7nx75--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5qcs2-l57tr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-5qcs2-l57tr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-45h85kz7nx75--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5qcs2-l57tr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2430848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4be43dc7-fd87-4e03-ace8-1bb6bd5bc1e3
correlation ID: 3db1a137-1e98-4c4a-af29-550f7818979f
client request ID: 15091261-7ca5-4fc4-9cba-cb6f5b7430f3Contributing tests (4)
Affected runs (6) | |||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 6 | 1.92%6 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 28: 0 · Oct 29: 5 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | INT | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-ph97cl"
Unexpected error:
<*errors.joinError | 0x118e0be62030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ph97cl-k2cv2-d9hk6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-ph97cl-k2cv2-d9hk6": nodes "cluster-np-ph97cl-k2cv2-d9hk6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x118e0b8bbd60>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ph97cl-k2cv2-d9hk6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ph97cl-k2cv2-d9hk6\": nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x118e0b8bbd40>{
msg: "failed to label node cluster-np-ph97cl-k2cv2-d9hk6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ph97cl-k2cv2-d9hk6\": nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x118e0b8bbd20>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ph97cl-k2cv2-d9hk6\": nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x118e0ba40b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-ph97cl-k2cv2-d9hk6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-ph97cl"
Unexpected error:
<*errors.joinError | 0x118e0be62030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ph97cl-k2cv2-d9hk6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-ph97cl-k2cv2-d9hk6": nodes "cluster-np-ph97cl-k2cv2-d9hk6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x118e0b8bbd60>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-ph97cl-k2cv2-d9hk6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ph97cl-k2cv2-d9hk6\": nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x118e0b8bbd40>{
msg: "failed to label node cluster-np-ph97cl-k2cv2-d9hk6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ph97cl-k2cv2-d9hk6\": nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x118e0b8bbd20>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-ph97cl-k2cv2-d9hk6\": nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x118e0ba40b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-ph97cl-k2cv2-d9hk6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-ph97cl-k2cv2-d9hk6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-8pc9jb"
Unexpected error:
<*errors.joinError | 0x19fbe63ec288>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-8pc9jb-xk8t4-5mzft": nodes "cluster-np-8pc9jb-xk8t4-5mzft" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x19fbe659e180>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e160>{
msg: "failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e140>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x19fbe658ea00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-8pc9jb-xk8t4-5mzft",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-8pc9jb"
Unexpected error:
<*errors.joinError | 0x19fbe63ec288>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-8pc9jb-xk8t4-5mzft": nodes "cluster-np-8pc9jb-xk8t4-5mzft" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x19fbe659e180>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e160>{
msg: "failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e140>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x19fbe658ea00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-8pc9jb-xk8t4-5mzft",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-spgbv9"
Unexpected error:
<*errors.joinError | 0xb85fe22660>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-spgbv9-bvtl9-fgbn6": nodes "cluster-np-spgbv9-bvtl9-fgbn6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xb8604081a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408180>{
msg: "failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408160>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xb860403860>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-spgbv9-bvtl9-fgbn6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-spgbv9"
Unexpected error:
<*errors.joinError | 0xb85fe22660>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-spgbv9-bvtl9-fgbn6": nodes "cluster-np-spgbv9-bvtl9-fgbn6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xb8604081a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408180>{
msg: "failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408160>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xb860403860>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-spgbv9-bvtl9-fgbn6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (6) | |||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 5 | 1.60%5 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 3 · Nov 3: 1 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x32bf764485a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
err: <*verifiers.pollObservationError | 0x32bf76448560>{
observation: <*errors.errorString | 0x32bf75b33060>{
s: "dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
},
interruption: <*fmt.wrapError | 0x32bf76448540>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x32bf74bea140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x32bf76448520>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x32bf76f2bb30>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x32bf764485a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zhhnjqb5pwp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
err: <*verifiers.pollObservationError | 0x32bf76448560>{
observation: <*errors.errorString | 0x32bf75b33060>{
s: "dev/centralus/ci01-j0707840-mgmt-2/ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster router=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003: expected matching summary after test start and within 8m; summaryUID=37d5ff2c-338b-4f58-b2ad-22912baf7e8f node=aks-userswft3-34297514-vmss000003 pass=b5c944cf-95e9-4664-9496-1197531a20a3 at=2026-11-03T11:51:34Z",
},
interruption: <*fmt.wrapError | 0x32bf76448540>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x32bf74bea140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x32bf76448520>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j0707840-mgmt-2\"; let ns = \"ocm-arohcpci01-2trnj0rr17e60i2gnvlvmp4u6rskijcf-cluster\"; let start = datetime(2026-11-03T11:37:01.2238296Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x32bf76f2bb30>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mhgz9kjj2rhp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2878f06ba220>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mhgz9kjj2rhp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j0858240-mgmt-1/ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster router=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004: expected matching summary after test start and within 8m; summaryUID=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004 pass=7f3ac93a-bffc-4454-bcb9-5c13ca80b1bc at=2026-11-03T00:35:13Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mhgz9kjj2rhp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j0858240-mgmt-1/ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster router=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004: expected matching summary after test start and within 8m; summaryUID=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004 pass=7f3ac93a-bffc-4454-bcb9-5c13ca80b1bc at=2026-11-03T00:35:13Z",
err: <*verifiers.pollObservationError | 0x2878f06ba200>{
observation: <*errors.errorString | 0x2878efd7f270>{
s: "dev/westus3/ci01-j0858240-mgmt-1/ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster router=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004: expected matching summary after test start and within 8m; summaryUID=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004 pass=7f3ac93a-bffc-4454-bcb9-5c13ca80b1bc at=2026-11-03T00:35:13Z",
},
interruption: <*fmt.wrapError | 0x2878f06ba1e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j0858240-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster\"; let start = datetime(2026-11-03T00:19:23.7510259Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2878f0494230>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2878f06ba1c0>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j0858240-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster\"; let start = datetime(2026-11-03T00:19:23.7510259Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2878f20100c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mhgz9kjj2rhp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2878f06ba220>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mhgz9kjj2rhp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j0858240-mgmt-1/ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster router=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004: expected matching summary after test start and within 8m; summaryUID=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004 pass=7f3ac93a-bffc-4454-bcb9-5c13ca80b1bc at=2026-11-03T00:35:13Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mhgz9kjj2rhp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j0858240-mgmt-1/ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster router=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004: expected matching summary after test start and within 8m; summaryUID=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004 pass=7f3ac93a-bffc-4454-bcb9-5c13ca80b1bc at=2026-11-03T00:35:13Z",
err: <*verifiers.pollObservationError | 0x2878f06ba200>{
observation: <*errors.errorString | 0x2878efd7f270>{
s: "dev/westus3/ci01-j0858240-mgmt-1/ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster router=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004: expected matching summary after test start and within 8m; summaryUID=884a36aa-43b1-49cd-849e-0a9f97c7f678 node=aks-userswft2-20285657-vmss000004 pass=7f3ac93a-bffc-4454-bcb9-5c13ca80b1bc at=2026-11-03T00:35:13Z",
},
interruption: <*fmt.wrapError | 0x2878f06ba1e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j0858240-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster\"; let start = datetime(2026-11-03T00:19:23.7510259Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2878f0494230>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2878f06ba1c0>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j0858240-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdlr774ff7plvv8r6kmkd288vpjt17-cluster\"; let start = datetime(2026-11-03T00:19:23.7510259Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2878f20100c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rp4xhjssqqxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x6f589abc5c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rp4xhjssqqxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9410688-mgmt-2/ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rp4xhjssqqxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9410688-mgmt-2/ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x6f589abc5a0>{
observation: <*errors.errorString | 0x6f5889a6bf0>{
s: "dev/westus3/ci01-j9410688-mgmt-2/ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x6f589abc580>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9410688-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster\"; let start = datetime(2026-11-02T16:55:25.5601113Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x6f588a5c1e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x6f589abc560>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9410688-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster\"; let start = datetime(2026-11-02T16:55:25.5601113Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x6f58a47f2f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rp4xhjssqqxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x6f589abc5c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rp4xhjssqqxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9410688-mgmt-2/ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rp4xhjssqqxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9410688-mgmt-2/ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x6f589abc5a0>{
observation: <*errors.errorString | 0x6f5889a6bf0>{
s: "dev/westus3/ci01-j9410688-mgmt-2/ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x6f589abc580>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9410688-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster\"; let start = datetime(2026-11-02T16:55:25.5601113Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x6f588a5c1e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x6f589abc560>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9410688-mgmt-2\"; let ns = \"ocm-arohcpci01-2tr75e93plphag4bt07h3tv9em2vffo1-cluster\"; let start = datetime(2026-11-02T16:55:25.5601113Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x6f58a47f2f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 5 | 1.60%5 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 4 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x21aad040a880>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
err: <*errors.errorString | 0x21aad0ac5810>{
s: "dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x21aad040a880>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-vr2q69pcmfwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
err: <*errors.errorString | 0x21aad0ac5810>{
s: "dev/centralus/ci01-j2113408-mgmt-1/ocm-arohcpci01-2trbgr5n3sjs3mg7fvbfajuvflmtttr2-cluster router=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001: expected matching summary after test start and within 8m; summaryUID=05a004e7-5e23-45ed-80bd-e94d3c0f312b node=aks-userswft1-48647740-vmss000001 pass=435417f8-40e0-4028-b4b5-8f472b09855f at=2026-11-02T22:07:50Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x29888a0e05e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x29888a0e05c0>{
observation: <*errors.errorString | 0x298889ba30a0>{
s: "dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x29888a0e05a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x298889b76320>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x29888a0e0580>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x29888b4d52f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x29888a0e05e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xvcbf4pn8n5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0x29888a0e05c0>{
observation: <*errors.errorString | 0x298889ba30a0>{
s: "dev/centralus/ci01-j8304128-mgmt-1/ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0x29888a0e05a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x298889b76320>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x29888a0e0580>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8304128-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9qigaekot5474ief116jvb69ippp6-cluster\"; let start = datetime(2026-11-02T19:57:04.9882676Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x29888b4d52f0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5q26tt22qqps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x11de573e86a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5q26tt22qqps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/canadacentral/ci01-j3603456-mgmt-1/ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5q26tt22qqps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/canadacentral/ci01-j3603456-mgmt-1/ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x11de573e8680>{
observation: <*errors.errorString | 0x11de57035400>{
s: "dev/canadacentral/ci01-j3603456-mgmt-1/ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x11de573e8660>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j3603456-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster\"; let start = datetime(2026-11-02T19:55:30.0703459Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x11de56118550>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x11de573e8640>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j3603456-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster\"; let start = datetime(2026-11-02T19:55:30.0703459Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x11de57357e00>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5q26tt22qqps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x11de573e86a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5q26tt22qqps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/canadacentral/ci01-j3603456-mgmt-1/ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5q26tt22qqps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/canadacentral/ci01-j3603456-mgmt-1/ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x11de573e8680>{
observation: <*errors.errorString | 0x11de57035400>{
s: "dev/canadacentral/ci01-j3603456-mgmt-1/ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x11de573e8660>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j3603456-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster\"; let start = datetime(2026-11-02T19:55:30.0703459Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x11de56118550>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x11de573e8640>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j3603456-mgmt-1\"; let ns = \"ocm-arohcpci01-2tr9psts61q9dn90u0sqk44jhh2l31sj-cluster\"; let start = datetime(2026-11-02T19:55:30.0703459Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x11de57357e00>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagementfull failure pattern: ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | alert | 4 | 1.28%4 of 313 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsOct 28: 4 · Oct 29: 2 · Oct 30: 4 · Oct 31: 0 · Nov 1: 2 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
}
}
--------------------------------------------------------------------------------alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
}
}
--------------------------------------------------------------------------------alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-dm/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-dm' could not be found."
}
}
--------------------------------------------------------------------------------Contributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 4 | 1.28%4 of 313 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 28: 0 · Oct 29: 3 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x3d12ed186260>:
DNS for route host agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3d12ecc524a0>{
msg: "DNS for agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x3d12ed186260>:
DNS for route host agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3d12ecc524a0>{
msg: "DNS for agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-xpvz7.apps.aro.cluster.4wmt6.j4847616.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-4g7vmt"
Unexpected error:
<*fmt.wrapError | 0x30c29e8a3000>:
DNS for route host agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x30c29ef369c0>{
msg: "DNS for agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-4g7vmt"
Unexpected error:
<*fmt.wrapError | 0x30c29e8a3000>:
DNS for route host agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x30c29ef369c0>{
msg: "DNS for agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-khwz2.apps.aro.cluster.88458.j0288256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x2e578d11a500>:
DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2e578d07c700>{
msg: "DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x2e578d11a500>:
DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2e578d07c700>{
msg: "DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (3)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insightsfull failure pattern: ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | alert | 4 | 1.28%4 of 313 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsOct 28: 2 · Oct 29: 1 · Oct 30: 2 · Oct 31: 0 · Nov 1: 1 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (2)alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
}
}
--------------------------------------------------------------------------------alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-dm: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-dm/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-dm' could not be found."
}
}
--------------------------------------------------------------------------------Contributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...full failure pattern: ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDBSignal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | provision | 4 | 1.28%4 of 313 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-02T22:48:17.015Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-02T22:48:18.483Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022?api-version=2021-04-01"
time=2026-11-02T22:48:20.632Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022?api-version=2021-04-01"
time=2026-11-02T22:48:24.763Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022?api-version=2021-04-01"
time=2026-11-02T22:48:34.398Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022?api-version=2021-04-01"
time=2026-11-02T22:48:52.441Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022?api-version=2021-04-01"
time=2026-11-02T22:49:26.226Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022?api-version=2021-04-01"
time=2026-11-02T22:49:26.226Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-02T22:49:26.865Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6556032%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbe5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022
time=2026-11-02T22:56:27.887Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/be5ac1899994ae7abefec359bf8bd40d5ceaa0bb93f842b43dfb6f8a8dd60022/operationStatuses/08584079471190040214\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8c291d3f-ec50-4882-be80-6cf71b316c5d\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6556032/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8c291d3f-ec50-4882-be80-6cf71b316c5d\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6556032/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j6556032\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 8339a6ee-faf4-4ea3-88ed-e67fea85a6da, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 6db162cf-b277-45c7-91c9-d0e752366f42, Request URI: /apps/56b56b41-1aa1-47de-9428-4a9b9ccbfcc7/services/9d38b9eb-92ab-49c7-bd48-ff5ea9e7ea22/partitions/2c82e19a-384e-4fa8-afcf-6cfbd77da7c7/replicas/134356927789909853s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-02T22:55:27.6763690Z, RequestEndTime: 2026-11-02T22:55:33.6617905Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T22:54:39.5490836Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.312,\\\\\\\"memory\\\\\\\":25823900.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0655,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":18},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T22:54:49.5503518Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.234,\\\\\\\"memory\\\\\\\":25821596.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0707,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":18},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T22:54:59.5659858Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.604,\\\\\\\"memory\\\\\\\":25839792.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0681,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":18},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T22:55:09.5664251Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.488,\\\\\\\"memory\\\\\\\":25839096.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0655,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":18},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T22:55:19.5820385Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.502,\\\\\\\"memory\\\\\\\":25827068.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0775,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":18},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T22:55:29.5826543Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.586,\\\\\\\"memory\\\\\\\":25831272.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0522,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":18}]}\\\\r\\\\nRequestStart: 2026-11-02T22:55:27.6763690Z; ResponseTime: 2026-11-02T22:55:33.6617905Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-centralus1-be312.documents.azure.com:11614/apps/56b56b41-1aa1-47de-9428-4a9b9ccbfcc7/services/9d38b9eb-92ab-49c7-bd48-ff5ea9e7ea22/partitions/2c82e19a-384e-4fa8-afcf-6cfbd77da7c7/replicas/134356927789909853s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5991.942, ActivityId: 6db162cf-b277-45c7-91c9-d0e752366f42, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11614 | status: Connected | lkt: 11/2/2026 10:55:33 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T22:55:27.6763690Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0022},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T22:55:27.6763712Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0061},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T22:55:27.6763773Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0854},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T22:55:27.6764627Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5992.376},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T22:55:33.6688387Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1109},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T22:55:33.6689496Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-02T22:53:14.3328635Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-02T22:53:14.3328635Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-02T22:53:19.6610114Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":871,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-02T17:06:27.099Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-02T17:06:28.457Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9?api-version=2021-04-01"
time=2026-11-02T17:06:30.645Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9?api-version=2021-04-01"
time=2026-11-02T17:06:34.782Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9?api-version=2021-04-01"
time=2026-11-02T17:06:43.650Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9?api-version=2021-04-01"
time=2026-11-02T17:07:00.806Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9?api-version=2021-04-01"
time=2026-11-02T17:07:33.147Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9?api-version=2021-04-01"
time=2026-11-02T17:07:33.147Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-02T17:07:34.062Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4084608%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fe1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9
time=2026-11-02T17:14:05.247Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/e1448273997cdd9603d86c3a11b5d1cd092fc61714f89ddaec6c85d5aadbc7b9/operationStatuses/08584079676320006943\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2ed2a6d5-3652-4e6c-8c52-0b85131fa7d2\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4084608/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2ed2a6d5-3652-4e6c-8c52-0b85131fa7d2\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4084608/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j4084608\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 08f0cf65-2348-491f-bd8b-d847ba7ebd1a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 9dc153dd-b804-41c6-9a92-68abaf4a2f8b, Request URI: /apps/3b245a36-dfbf-44c5-8156-a0a45e8e24b5/services/6a3e668c-4dc7-4354-a8d0-019371c578a1/partitions/4792561f-28fd-4113-ab39-553cacd248ac/replicas/134356458617870525s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-02T17:13:19.5588958Z, RequestEndTime: 2026-11-02T17:13:25.2161598Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T17:12:31.1012714Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.092,\\\\\\\"memory\\\\\\\":26173580.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1508,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":151},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T17:12:41.1178158Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.956,\\\\\\\"memory\\\\\\\":26180888.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.9743,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":151},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T17:12:51.1336434Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.409,\\\\\\\"memory\\\\\\\":26187336.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1032,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":151},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T17:13:01.1492613Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.995,\\\\\\\"memory\\\\\\\":26191572.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0947,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":151},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T17:13:11.1509202Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.500,\\\\\\\"memory\\\\\\\":26176356.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1182,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":151},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-02T17:13:21.1526381Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.344,\\\\\\\"memory\\\\\\\":26201352.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0927,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":151}]}\\\\r\\\\nRequestStart: 2026-11-02T17:13:19.5588958Z; ResponseTime: 2026-11-02T17:13:25.2161598Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be187.documents.azure.com:11914/apps/3b245a36-dfbf-44c5-8156-a0a45e8e24b5/services/6a3e668c-4dc7-4354-a8d0-019371c578a1/partitions/4792561f-28fd-4113-ab39-553cacd248ac/replicas/134356458617870525s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5652.576, ActivityId: 9dc153dd-b804-41c6-9a92-68abaf4a2f8b, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11914 | status: Connected | lkt: 11/2/2026 5:13:25 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T17:13:19.5588958Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0211},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T17:13:19.5589169Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0098},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T17:13:19.5589267Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.146},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T17:13:19.5590727Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5653.1755},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T17:13:25.2122482Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1462},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-02T17:13:25.2123944Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-02T17:11:05.5591773Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-02T17:11:05.5591773Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-02T17:11:05.5904818Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":871,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-30T14:18:12.395Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-30T14:18:14.075Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-30T14:18:15.333Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9737a4696ad51ddffdeecc01ed7b6295a0e57b14d828e8e7be1b4220fb612ba0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2776960%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9737a4696ad51ddffdeecc01ed7b6295a0e57b14d828e8e7be1b4220fb612ba0
time=2026-10-30T14:25:46.553Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2776960/providers/Microsoft.Resources/deployments/9737a4696ad51ddffdeecc01ed7b6295a0e57b14d828e8e7be1b4220fb612ba0/operationStatuses/08584082369908647509\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2c4ba16f-a7c7-4c02-8751-7db85f903767\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2776960/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2c4ba16f-a7c7-4c02-8751-7db85f903767\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2776960/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j2776960\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 6c580d55-785f-4d91-b9af-37709afd1c9a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 0ea3984e-666b-4a2b-a5e8-a930ddf4631d, Request URI: /apps/af11721d-6fc8-46aa-8f69-d3f60d22671a/services/fa30e1ef-2839-49f2-91ea-6f4384e7c41e/partitions/18a37c77-6dbe-4c8f-8e31-306da959eea2/replicas/134371044267353265s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-10-30T14:23:58.0932729Z, RequestEndTime: 2026-10-30T14:24:03.7028220Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-30T14:23:05.5525763Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.268,\\\\\\\"memory\\\\\\\":22690356.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0634,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":169},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-30T14:23:15.5547365Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.126,\\\\\\\"memory\\\\\\\":22693852.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0601,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":169},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-30T14:23:25.5569495Z\\\\\\\",\\\\\\\"cpu\\\\\\\":3.516,\\\\\\\"memory\\\\\\\":22673424.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0817,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":168},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-30T14:23:35.5592306Z\\\\\\\",\\\\\\\"cpu\\\\\\\":3.242,\\\\\\\"memory\\\\\\\":22653140.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0644,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":170},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-30T14:23:45.5610572Z\\\\\\\",\\\\\\\"cpu\\\\\\\":5.605,\\\\\\\"memory\\\\\\\":22668712.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1065,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":173},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-30T14:23:55.5619854Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.523,\\\\\\\"memory\\\\\\\":22681072.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0659,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":176}]}\\\\r\\\\nRequestStart: 2026-10-30T14:23:58.0932729Z; ResponseTime: 2026-10-30T14:24:03.7028220Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be149.documents.azure.com:11011/apps/af11721d-6fc8-46aa-8f69-d3f60d22671a/services/fa30e1ef-2839-49f2-91ea-6f4384e7c41e/partitions/18a37c77-6dbe-4c8f-8e31-306da959eea2/replicas/134371044267353265s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5605.075, ActivityId: 0ea3984e-666b-4a2b-a5e8-a930ddf4631d, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11011 | status: Connected | lkt: 10/30/2026 2:24:03 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-30T14:23:58.0932729Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0038},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-30T14:23:58.0932767Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0116},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-30T14:23:58.0932883Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1186},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-30T14:23:58.0934069Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5606.0773},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-30T14:24:03.6994842Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1559},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-30T14:24:03.6996401Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-10-30T14:21:44.1040158Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-10-30T14:21:44.1040158Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-10-30T14:21:48.7005618Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":879,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 4 | 1.28%4 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 28: 0 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 2 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3a3b4c2860c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
err: <*verifiers.pollObservationError | 0x3a3b4c2860a0>{
observation: <*errors.errorString | 0x3a3b4ae007e0>{
s: "dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
},
interruption: <*fmt.wrapError | 0x3a3b4c286080>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3a3b4b2bc140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3a3b4c286060>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3a3b4cc54210>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3a3b4c2860c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5mq8qj7r5z5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
err: <*verifiers.pollObservationError | 0x3a3b4c2860a0>{
observation: <*errors.errorString | 0x3a3b4ae007e0>{
s: "dev/centralus/ci01-j5985664-mgmt-1/ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster router=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001: expected matching summary after test start and within 8m; summaryUID=90ab3eca-a11b-4bd6-ad1c-c586c30e4714 node=aks-userswft1-56461193-vmss000001 pass=c620d370-285c-4105-a68a-15b3af51c200 at=2026-11-03T00:32:32Z",
},
interruption: <*fmt.wrapError | 0x3a3b4c286080>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3a3b4b2bc140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3a3b4c286060>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j5985664-mgmt-1\"; let ns = \"ocm-arohcpci01-2trdkknmfi0cmhb1a4nkohkjgrp69dn9-cluster\"; let start = datetime(2026-11-03T00:17:40.5789805Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3a3b4cc54210>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rm4qc5cbrxrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1609bc5dad80>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rm4qc5cbrxrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6567680-mgmt-2/ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster router=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004: expected matching summary after test start and within 8m; summaryUID=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004 pass=ac7a7aff-3eba-4909-8ed7-180252914bb6 at=2026-11-03T00:10:15Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rm4qc5cbrxrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6567680-mgmt-2/ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster router=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004: expected matching summary after test start and within 8m; summaryUID=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004 pass=ac7a7aff-3eba-4909-8ed7-180252914bb6 at=2026-11-03T00:10:15Z",
err: <*verifiers.pollObservationError | 0x1609bc5dad60>{
observation: <*errors.errorString | 0x1609bb49b6e0>{
s: "dev/canadacentral/ci01-j6567680-mgmt-2/ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster router=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004: expected matching summary after test start and within 8m; summaryUID=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004 pass=ac7a7aff-3eba-4909-8ed7-180252914bb6 at=2026-11-03T00:10:15Z",
},
interruption: <*fmt.wrapError | 0x1609bc5dad40>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j6567680-mgmt-2\"; let ns = \"ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster\"; let start = datetime(2026-11-02T23:44:18.2611774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1609bbcba640>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1609bc5dad20>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j6567680-mgmt-2\"; let ns = \"ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster\"; let start = datetime(2026-11-02T23:44:18.2611774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1609bce19260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rm4qc5cbrxrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1609bc5dad80>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rm4qc5cbrxrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6567680-mgmt-2/ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster router=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004: expected matching summary after test start and within 8m; summaryUID=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004 pass=ac7a7aff-3eba-4909-8ed7-180252914bb6 at=2026-11-03T00:10:15Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rm4qc5cbrxrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6567680-mgmt-2/ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster router=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004: expected matching summary after test start and within 8m; summaryUID=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004 pass=ac7a7aff-3eba-4909-8ed7-180252914bb6 at=2026-11-03T00:10:15Z",
err: <*verifiers.pollObservationError | 0x1609bc5dad60>{
observation: <*errors.errorString | 0x1609bb49b6e0>{
s: "dev/canadacentral/ci01-j6567680-mgmt-2/ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster router=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004: expected matching summary after test start and within 8m; summaryUID=004cc18d-acc5-474b-925f-0b90d698898e node=aks-userswft2-32072556-vmss000004 pass=ac7a7aff-3eba-4909-8ed7-180252914bb6 at=2026-11-03T00:10:15Z",
},
interruption: <*fmt.wrapError | 0x1609bc5dad40>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j6567680-mgmt-2\"; let ns = \"ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster\"; let start = datetime(2026-11-02T23:44:18.2611774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1609bbcba640>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1609bc5dad20>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j6567680-mgmt-2\"; let ns = \"ocm-arohcpci01-2trd4vt2g4nanscoppcbjsmc0p5h8bdo-cluster\"; let start = datetime(2026-11-02T23:44:18.2611774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1609bce19260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rdh57bhk9b9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x14f73874320>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rdh57bhk9b9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j3760640-mgmt-2/ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster router=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003: expected matching summary after test start and within 8m; summaryUID=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003 pass=ba9f824f-9d65-4404-8922-128fb381654d at=2026-10-29T22:05:14Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rdh57bhk9b9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j3760640-mgmt-2/ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster router=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003: expected matching summary after test start and within 8m; summaryUID=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003 pass=ba9f824f-9d65-4404-8922-128fb381654d at=2026-10-29T22:05:14Z",
err: <*verifiers.pollObservationError | 0x14f73874300>{
observation: <*errors.errorString | 0x14f72a80360>{
s: "dev/centralus/ci01-j3760640-mgmt-2/ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster router=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003: expected matching summary after test start and within 8m; summaryUID=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003 pass=ba9f824f-9d65-4404-8922-128fb381654d at=2026-10-29T22:05:14Z",
},
interruption: <*fmt.wrapError | 0x14f738742e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j3760640-mgmt-2\"; let ns = \"ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster\"; let start = datetime(2026-10-29T21:38:05.9291513Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x14f72625590>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x14f738742c0>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j3760640-mgmt-2\"; let ns = \"ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster\"; let start = datetime(2026-10-29T21:38:05.9291513Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x14f7486c540>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rdh57bhk9b9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x14f73874320>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rdh57bhk9b9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j3760640-mgmt-2/ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster router=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003: expected matching summary after test start and within 8m; summaryUID=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003 pass=ba9f824f-9d65-4404-8922-128fb381654d at=2026-10-29T22:05:14Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rdh57bhk9b9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j3760640-mgmt-2/ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster router=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003: expected matching summary after test start and within 8m; summaryUID=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003 pass=ba9f824f-9d65-4404-8922-128fb381654d at=2026-10-29T22:05:14Z",
err: <*verifiers.pollObservationError | 0x14f73874300>{
observation: <*errors.errorString | 0x14f72a80360>{
s: "dev/centralus/ci01-j3760640-mgmt-2/ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster router=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003: expected matching summary after test start and within 8m; summaryUID=00945b06-bc1d-4ed1-9b14-2307674f68ba node=aks-userswft1-23304135-vmss000003 pass=ba9f824f-9d65-4404-8922-128fb381654d at=2026-10-29T22:05:14Z",
},
interruption: <*fmt.wrapError | 0x14f738742e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j3760640-mgmt-2\"; let ns = \"ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster\"; let start = datetime(2026-10-29T21:38:05.9291513Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x14f72625590>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x14f738742c0>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j3760640-mgmt-2\"; let ns = \"ocm-arohcpci01-2tomu2t3n886vc45aigsj4l6cpdmi0t2-cluster\"; let start = datetime(2026-10-29T21:38:05.9291513Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x14f7486c540>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
alert [svc] hypershiftOperatorPodsDown firedfull failure pattern: alert [svc] hypershiftOperatorPodsDown fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | alert | 4 | 1.28%4 of 313 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 28: 1 · Oct 29: 4 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:11:35Z Ended: 2026-10-29T16:17:37Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j8756352-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j8756352-mgmt-2 alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T14:55:08Z Ended: 2026-10-29T15:01:06Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j0084992-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j0084992-mgmt-2 alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T12:14:15Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j2007552-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j2007552-mgmt-2 Contributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||
Environment: INT
E2E Jobs
36
Success Rate
80.56%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.100.00% (29/29)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 80.00% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster then delete it by deleting the customer resource group | |
| 84.44% | 45 | integration/parallel | Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully | |
| 84.44% | 45 | integration/parallel | Customer should be able to update node pool labels and taints |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 2 | 5.56%2 of 36 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 12 · Nov 4: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x20336c4ed070>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x20336c4ed070>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3c65f081b070>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3c65f081b070>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.008s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2d204ffaa910>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.008s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2d204ffaa910>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 2.78%1 of 36 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sl7v6jcfzpzn/cluster-candidate-5-0-rw67x7 should provision
Unexpected error:
<*fmt.wrapError | 0x320a55350300>:
failed to create HCP cluster cluster-candidate-5-0-rw67x7: failed waiting for cluster="cluster-candidate-5-0-rw67x7" in resourcegroup="rg-candidate-5-0-sl7v6jcfzpzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"name": "e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"status": "Failed",
"startTime": "2026-10-29T01:24:15.287612604Z",
"endTime": "2026-10-29T01:53:18.05722195Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
correlation ID: c2102baf-9a88-4da3-a6f8-854193b8aaa9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sl7v6jcfzpzn/cluster-candidate-5-0-rw67x7 should provision
Unexpected error:
<*fmt.wrapError | 0x320a55350300>:
failed to create HCP cluster cluster-candidate-5-0-rw67x7: failed waiting for cluster="cluster-candidate-5-0-rw67x7" in resourcegroup="rg-candidate-5-0-sl7v6jcfzpzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"name": "e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"status": "Failed",
"startTime": "2026-10-29T01:24:15.287612604Z",
"endTime": "2026-10-29T01:53:18.05722195Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
correlation ID: c2102baf-9a88-4da3-a6f8-854193b8aaa9
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 2.78%1 of 36 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x22ca12a3a120>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-c8zn4dkvb2j2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad454da8-1cc3-454a-97b0-d989f045c80d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad454da8-1cc3-454a-97b0-d989f045c80d",
"name": "ad454da8-1cc3-454a-97b0-d989f045c80d",
"status": "Failed",
"startTime": "2026-10-31T00:53:10.070017502Z",
"endTime": "2026-10-31T01:22:15.104851715Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 22m26s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad454da8-1cc3-454a-97b0-d989f045c80d
correlation ID: 5729793c-88a4-4201-a412-233020f32e1f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x22ca12a3a120>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-c8zn4dkvb2j2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad454da8-1cc3-454a-97b0-d989f045c80d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad454da8-1cc3-454a-97b0-d989f045c80d",
"name": "ad454da8-1cc3-454a-97b0-d989f045c80d",
"status": "Failed",
"startTime": "2026-10-31T00:53:10.070017502Z",
"endTime": "2026-10-31T01:22:15.104851715Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 22m26s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad454da8-1cc3-454a-97b0-d989f045c80d
correlation ID: 5729793c-88a4-4201-a412-233020f32e1f
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||
OIDC username should use the platform-selected prefix after the policy updatefull failure pattern: OIDC username should use the platform-selected prefix after the policy update Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.78%1 of 36 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:291]: Timed out after 300.001s.
OIDC username should use the platform-selected prefix after the policy update
Expected
<string>: prefix-8d78d528-259e-4079-aa43-ea08b66a56b3
not to equal
<string>: prefix-8d78d528-259e-4079-aa43-ea08b66a56b3
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:291]: Timed out after 300.001s.
OIDC username should use the platform-selected prefix after the policy update
Expected
<string>: prefix-8d78d528-259e-4079-aa43-ea08b66a56b3
not to equal
<string>: prefix-8d78d528-259e-4079-aa43-ea08b66a56b3Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.78%1 of 36 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-10-29T19:59:31Z"}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.78%1 of 36 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | DEV | |||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x866rm"
Unexpected error:
<*errors.joinError | 0x2cfb1b0a33e0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-x866rm-lmp2b-9wggz": nodes "cluster-np-x866rm-lmp2b-9wggz" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2cfb1c1b05e0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05c0>{
msg: "failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05a0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2cfb1c1b6460>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-x866rm-lmp2b-9wggz",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x866rm"
Unexpected error:
<*errors.joinError | 0x2cfb1b0a33e0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-x866rm-lmp2b-9wggz": nodes "cluster-np-x866rm-lmp2b-9wggz" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2cfb1c1b05e0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05c0>{
msg: "failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05a0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2cfb1c1b6460>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-x866rm-lmp2b-9wggz",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||
Environment: STG
E2E Jobs
30
Success Rate
33.33%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.76.92% (10/13)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
--------------------------------------------------------------------------------full failure pattern: -------------------------------------------------------------------------------- Signal: NoiseSignal: Noise — mostly punctuation | unknown | 1 | 3.33%1 of 30 job runs affected | NoiseSignal: Noise — mostly punctuation | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-22x7k2t2rltt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c9f8ca64-5055-415e-8afb-a77f96fe9abd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c9f8ca64-5055-415e-8afb-a77f96fe9abd",
"name": "c9f8ca64-5055-415e-8afb-a77f96fe9abd",
"status": "Failed",
"startTime": "2026-10-30T19:36:57.425149813Z",
"endTime": "2026-10-30T19:56:03.720376948Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine cluster-np-one-node-w5fb6-f4qtp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine cluster-np-one-node-w5fb6-f4qtp: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
]
Expected
<[]error | len:1, cap:1>: [
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c9f8ca64-5055-415e-8afb-a77f96fe9abd
correlation ID: b5c8ad2d-17ee-49a6-bbf1-5702758258ea
client request ID: unavailable,
]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-22x7k2t2rltt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c9f8ca64-5055-415e-8afb-a77f96fe9abd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c9f8ca64-5055-415e-8afb-a77f96fe9abd",
"name": "c9f8ca64-5055-415e-8afb-a77f96fe9abd",
"status": "Failed",
"startTime": "2026-10-30T19:36:57.425149813Z",
"endTime": "2026-10-30T19:56:03.720376948Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine cluster-np-one-node-w5fb6-f4qtp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine cluster-np-one-node-w5fb6-f4qtp: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
]
Expected
<[]error | len:1, cap:1>: [
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c9f8ca64-5055-415e-8afb-a77f96fe9abd
correlation ID: b5c8ad2d-17ee-49a6-bbf1-5702758258ea
client request ID: unavailable,
]
to be emptyContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpen...full failure pattern: ERROR CODE: GatewayTimeout; detail message The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 25 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
<*fmt.wrapError | 0x3b12782929c0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-gsj2rbf9gl9w" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
correlation ID: 9bc223c2-995a-427c-949e-62ca384432ff
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
<*fmt.wrapError | 0x3b12782929c0>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-gsj2rbf9gl9w" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24866a5c-cc1c-4784-a317-3d5ad0edc109
correlation ID: 9bc223c2-995a-427c-949e-62ca384432ff
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x2b292c1c0168>:
resource group "rg-cordon-test-gr9c8pd6vnnm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gr9c8pd6vnnm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
correlation ID: c29bc14e-8e04-4ca1-9e28-44104dedab13
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x2b292c1c0168>:
resource group "rg-cordon-test-gr9c8pd6vnnm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gr9c8pd6vnnm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c92de6c-642f-419d-9f8a-c1b9129fd432
correlation ID: c29bc14e-8e04-4ca1-9e28-44104dedab13
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0xe5b8ee500a8>:
resource group "clusternp128-br4ctdhmf9nj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-br4ctdhmf9nj" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
correlation ID: 93d49f6e-4f14-4abc-8522-d90532e9b6a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0xe5b8ee500a8>:
resource group "clusternp128-br4ctdhmf9nj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-br4ctdhmf9nj" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
--------------------------------------------------------------------------------
RESPONSE 504: 504 Gateway Timeout
ERROR CODE: GatewayTimeout
--------------------------------------------------------------------------------
{
"error": {
"code": "GatewayTimeout",
"message": "The gateway did not receive a response from 'Microsoft.RedHatOpenShift' within the specified time period."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/103d4c16-12c2-4ea8-804d-24535d4fc81a
correlation ID: 93d49f6e-4f14-4abc-8522-d90532e9b6a7
client request ID: unavailable
occurredContributing tests (25)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 12 · Nov 4: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e0c4e911a60>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e0c4e911a60>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x145f6ff3e730>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x145f6ff3e730>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x34f47ff7360>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x34f47ff7360>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 12 · Nov 4: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x293841bddd80>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x293841bddd80>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2ae0de368e10>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2ae0de368e10>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xdcc5690bb80>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xdcc5690bb80>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x2db751ad0e20>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-l6b4xb5plzmd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d2fac81b-dabd-49fb-aa25-c6fae7fb122b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d2fac81b-dabd-49fb-aa25-c6fae7fb122b",
"name": "d2fac81b-dabd-49fb-aa25-c6fae7fb122b",
"status": "Failed",
"startTime": "2026-10-30T19:32:54.516822894Z",
"endTime": "2026-10-30T19:52:03.747155761Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-cordon-np-7rwx7-6sclm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-cordon-np-7rwx7-vcwln: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cordon-np-7rwx7-6sclm: Creating: virtualmachine creating or updating\nMachine cluster-cordon-np-7rwx7-vcwln: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d2fac81b-dabd-49fb-aa25-c6fae7fb122b
correlation ID: 2401afbe-64cf-4fb2-9352-e6078826e657
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x2db751ad0e20>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-l6b4xb5plzmd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d2fac81b-dabd-49fb-aa25-c6fae7fb122b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d2fac81b-dabd-49fb-aa25-c6fae7fb122b",
"name": "d2fac81b-dabd-49fb-aa25-c6fae7fb122b",
"status": "Failed",
"startTime": "2026-10-30T19:32:54.516822894Z",
"endTime": "2026-10-30T19:52:03.747155761Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-cordon-np-7rwx7-6sclm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-cordon-np-7rwx7-vcwln: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cordon-np-7rwx7-6sclm: Creating: virtualmachine creating or updating\nMachine cluster-cordon-np-7rwx7-vcwln: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d2fac81b-dabd-49fb-aa25-c6fae7fb122b
correlation ID: 2401afbe-64cf-4fb2-9352-e6078826e657
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2b898ea49840>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-ml4vb7" in resourcegroup="rg-np-version-upgrade-ml4vb7-cnm9xh99w75q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/141c3999-2d3d-49ae-8633-d433af93c554
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/141c3999-2d3d-49ae-8633-d433af93c554",
"name": "141c3999-2d3d-49ae-8633-d433af93c554",
"status": "Failed",
"startTime": "2026-10-30T19:37:37.869979523Z",
"endTime": "2026-10-30T19:56:44.005517219Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-w2vbk-42czv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npupgrade-4-20-w2vbk-xmxd2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-w2vbk-42czv: Creating: virtualmachine creating or updating\nMachine cluster-npupgrade-4-20-w2vbk-xmxd2: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/141c3999-2d3d-49ae-8633-d433af93c554
correlation ID: 5749dda1-6fb9-463d-aa08-6f276b220d61
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2b898ea49840>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-ml4vb7" in resourcegroup="rg-np-version-upgrade-ml4vb7-cnm9xh99w75q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/141c3999-2d3d-49ae-8633-d433af93c554
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/141c3999-2d3d-49ae-8633-d433af93c554",
"name": "141c3999-2d3d-49ae-8633-d433af93c554",
"status": "Failed",
"startTime": "2026-10-30T19:37:37.869979523Z",
"endTime": "2026-10-30T19:56:44.005517219Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-w2vbk-42czv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npupgrade-4-20-w2vbk-xmxd2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-w2vbk-42czv: Creating: virtualmachine creating or updating\nMachine cluster-npupgrade-4-20-w2vbk-xmxd2: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/141c3999-2d3d-49ae-8633-d433af93c554
correlation ID: 5749dda1-6fb9-463d-aa08-6f276b220d61
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: ProviderError; detail message Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.; provider Microsoft.R...full failure pattern: ERROR CODE: ProviderError; detail message Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 2 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x31106c3400a8>:
resource group "rg-candidate-4-23-8g66dmqqdl2t": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-candidate-4-23-8g66dmqqdl2t": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: cdec6720-cba1-4de4-b58b-ba3824034d6e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x31106c3400a8>:
resource group "rg-candidate-4-23-8g66dmqqdl2t": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-candidate-4-23-8g66dmqqdl2t": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-8g66dmqqdl2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: cdec6720-cba1-4de4-b58b-ba3824034d6e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3509f2bfe0f0>:
resource group "autoscaling-cluster-s7nqf86dlmwk": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="autoscaling-cluster-s7nqf86dlmwk": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: c0db6013-b086-440f-864e-95153120e95f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353]: Unexpected error:
<*errors.joinError | 0x3509f2bfe0f0>:
resource group "autoscaling-cluster-s7nqf86dlmwk": failed to cleanup resource group: failed listing hcp clusters in resourcegroup="autoscaling-cluster-s7nqf86dlmwk": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE: ProviderError
--------------------------------------------------------------------------------
{
"error": {
"code": "ProviderError",
"message": "Resource provider 'Microsoft.RedHatOpenShift' failed to return collection response for type 'HCPOpenShiftClusters'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-s7nqf86dlmwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: c0db6013-b086-440f-864e-95153120e95f
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x35318b3701a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-qgkvdgwhnvp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"name": "f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"status": "Failed",
"startTime": "2026-11-03T03:30:50.28074301Z",
"endTime": "2026-11-03T03:49:59.74254364Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
correlation ID: 414909e2-b5b8-45c7-b8ee-51a914e3f24f
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x35318b3701a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-qgkvdgwhnvp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"name": "f42a0060-7877-452e-8e48-c9e9d3cd3ebc",
"status": "Failed",
"startTime": "2026-11-03T03:30:50.28074301Z",
"endTime": "2026-11-03T03:49:59.74254364Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f42a0060-7877-452e-8e48-c9e9d3cd3ebc
correlation ID: 414909e2-b5b8-45c7-b8ee-51a914e3f24f
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: ConflictingConcurrentWriteNotAllowed; detail message The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later.; provider ...full failure pattern: ERROR CODE: ConflictingConcurrentWriteNotAllowed; detail message The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
<*exported.ResponseError | 0x3c8f7d214900>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ConflictingConcurrentWriteNotAllowed
--------------------------------------------------------------------------------
{
"error": {
"code": "ConflictingConcurrentWriteNotAllowed",
"message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
correlation ID: fbbd5005-0e31-417c-aa2b-b969b8f7ab06
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
<*exported.ResponseError | 0x3c8f7d214900>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
--------------------------------------------------------------------------------
RESPONSE 409: 409 Conflict
ERROR CODE: ConflictingConcurrentWriteNotAllowed
--------------------------------------------------------------------------------
{
"error": {
"code": "ConflictingConcurrentWriteNotAllowed",
"message": "The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v85mp4jgx4xv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
correlation ID: fbbd5005-0e31-417c-aa2b-b969b8f7ab06
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, cluster-image-registry-operator, cluster-network-operator, dns-operator, hosted-cluster-config-operator, ingress-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-vjvvwt6d4mf7/cluster-candidate-5-1-t5ntfs should provision
Unexpected error:
<*fmt.wrapError | 0x18ecee674de0>:
failed to create HCP cluster cluster-candidate-5-1-t5ntfs: failed waiting for cluster="cluster-candidate-5-1-t5ntfs" in resourcegroup="rg-candidate-5-1-vjvvwt6d4mf7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"name": "8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"status": "Failed",
"startTime": "2026-11-03T10:01:43.423478369Z",
"endTime": "2026-11-03T10:30:49.760962704Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, ingress-operator, catalog-operator, hosted-cluster-config-operator, cluster-network-operator, cluster-image-registry-operator, packageserver, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
correlation ID: d9c0e94e-e861-412f-a3b2-f93dbcd093b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-vjvvwt6d4mf7/cluster-candidate-5-1-t5ntfs should provision
Unexpected error:
<*fmt.wrapError | 0x18ecee674de0>:
failed to create HCP cluster cluster-candidate-5-1-t5ntfs: failed waiting for cluster="cluster-candidate-5-1-t5ntfs" in resourcegroup="rg-candidate-5-1-vjvvwt6d4mf7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"name": "8dcb2c41-ae04-4687-bcc7-542c0049d2db",
"status": "Failed",
"startTime": "2026-11-03T10:01:43.423478369Z",
"endTime": "2026-11-03T10:30:49.760962704Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, ingress-operator, catalog-operator, hosted-cluster-config-operator, cluster-network-operator, cluster-image-registry-operator, packageserver, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dcb2c41-ae04-4687-bcc7-542c0049d2db
correlation ID: d9c0e94e-e861-412f-a3b2-f93dbcd093b1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x2486b0990c0>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fhxl6psjw7xx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9",
"name": "23e1c59e-d85b-40a6-b315-78ca430171e9",
"status": "Failed",
"startTime": "2026-11-03T03:31:25.980900033Z",
"endTime": "2026-11-03T03:50:29.670327156Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
correlation ID: 7d147fcf-e938-458b-891d-77c7c176f302
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x2486b0990c0>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fhxl6psjw7xx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9",
"name": "23e1c59e-d85b-40a6-b315-78ca430171e9",
"status": "Failed",
"startTime": "2026-11-03T03:31:25.980900033Z",
"endTime": "2026-11-03T03:50:29.670327156Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23e1c59e-d85b-40a6-b315-78ca430171e9
correlation ID: 7d147fcf-e938-458b-891d-77c7c176f302
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidResource; detail message ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions.; provider Mic...full failure pattern: ERROR CODE: InvalidResource; detail message ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 1 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x2e79db1b2e0>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-22mskvfrsnk8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InvalidResource
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936",
"name": "f4b8761b-d251-4636-b586-532f30b02936",
"status": "Failed",
"startTime": "2026-11-03T07:12:43.456403262Z",
"endTime": "2026-11-03T07:23:57.559738554Z",
"error": {
"code": "InvalidResource",
"message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
correlation ID: 3507dcf7-3522-4e9c-bdf1-375d292a8d55
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x2e79db1b2e0>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-22mskvfrsnk8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InvalidResource
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936",
"name": "f4b8761b-d251-4636-b586-532f30b02936",
"status": "Failed",
"startTime": "2026-11-03T07:12:43.456403262Z",
"endTime": "2026-11-03T07:23:57.559738554Z",
"error": {
"code": "InvalidResource",
"message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4b8761b-d251-4636-b586-532f30b02936
correlation ID: 3507dcf7-3522-4e9c-bdf1-375d292a8d55
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-02T09:54:16Z"}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-<id>])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-<id>]) Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.33%1 of 30 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:886]: Timed out after 900.004s.
node pool np-1 never detached the day 2 MI /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/acr-pull-2pg5nqg2c8wm/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi-2 after removal
Expected success, but got an error:
<*errors.errorString | 0x18a468a961b0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:886]: Timed out after 900.004s.
node pool np-1 never detached the day 2 MI /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/acr-pull-2pg5nqg2c8wm/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi-2 after removal
Expected success, but got an error:
<*errors.errorString | 0x18a468a961b0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
E2E Jobs
52
Success Rate
59.62%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.72.09% (31/43)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 83.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 91.67% | 12 | prod/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration | |
| 91.67% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 91.67% | 12 | prod/parallel | Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 4 | 7.69%4 of 52 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 28: 19 · Oct 29: 3 · Oct 30: 1 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x629f8415040>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-rpcrpkkmh9m5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/bea08509-d3b8-4280-9d7c-4a8d7e36de6f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/bea08509-d3b8-4280-9d7c-4a8d7e36de6f",
"name": "bea08509-d3b8-4280-9d7c-4a8d7e36de6f",
"status": "Failed",
"startTime": "2026-10-30T03:49:50.956783737Z",
"endTime": "2026-10-30T04:18:57.343412876Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/bea08509-d3b8-4280-9d7c-4a8d7e36de6f
correlation ID: ed76b30e-4724-433e-9a81-5ae208b72608
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x629f8415040>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-rpcrpkkmh9m5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/bea08509-d3b8-4280-9d7c-4a8d7e36de6f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/bea08509-d3b8-4280-9d7c-4a8d7e36de6f",
"name": "bea08509-d3b8-4280-9d7c-4a8d7e36de6f",
"status": "Failed",
"startTime": "2026-10-30T03:49:50.956783737Z",
"endTime": "2026-10-30T04:18:57.343412876Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/bea08509-d3b8-4280-9d7c-4a8d7e36de6f
correlation ID: ed76b30e-4724-433e-9a81-5ae208b72608
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x1def056ca980>:
failed to create HCP cluster ea-listzz8ctk: failed waiting for cluster="ea-listzz8ctk" in resourcegroup="ea-list-4jljqltdpf22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51",
"name": "671f379d-8472-4150-82a8-d71efe3bef51",
"status": "Failed",
"startTime": "2026-10-29T22:40:05.142063562Z",
"endTime": "2026-10-29T23:09:09.015450129Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
correlation ID: 462ab43f-575c-42b1-ab0d-20c41a4d0046
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x1def056ca980>:
failed to create HCP cluster ea-listzz8ctk: failed waiting for cluster="ea-listzz8ctk" in resourcegroup="ea-list-4jljqltdpf22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51",
"name": "671f379d-8472-4150-82a8-d71efe3bef51",
"status": "Failed",
"startTime": "2026-10-29T22:40:05.142063562Z",
"endTime": "2026-10-29T23:09:09.015450129Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
correlation ID: 462ab43f-575c-42b1-ab0d-20c41a4d0046
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x241c81f32b80>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-58txd54zlccz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"name": "c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"status": "Failed",
"startTime": "2026-10-29T14:34:25.586865421Z",
"endTime": "2026-10-29T15:03:34.559330088Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
correlation ID: 5c469ba9-6c92-4dd6-9ba5-04a68fee9ef5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x241c81f32b80>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-58txd54zlccz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"name": "c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"status": "Failed",
"startTime": "2026-10-29T14:34:25.586865421Z",
"endTime": "2026-10-29T15:03:34.559330088Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
correlation ID: 5c469ba9-6c92-4dd6-9ba5-04a68fee9ef5
client request ID: unavailable
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 4 | 7.69%4 of 52 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 28: 0 · Oct 29: 2 · Oct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x100093d12a00>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-nlwwcc4hsgpp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/7beab753-b75d-4317-bea5-cd144607299f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/7beab753-b75d-4317-bea5-cd144607299f",
"name": "7beab753-b75d-4317-bea5-cd144607299f",
"status": "Failed",
"startTime": "2026-10-30T22:55:16.155391602Z",
"endTime": "2026-10-30T23:14:20.238434762Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/7beab753-b75d-4317-bea5-cd144607299f
correlation ID: faee6cbd-33e5-446e-9d93-e6c2de3830ab
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x100093d12a00>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-nlwwcc4hsgpp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/7beab753-b75d-4317-bea5-cd144607299f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/7beab753-b75d-4317-bea5-cd144607299f",
"name": "7beab753-b75d-4317-bea5-cd144607299f",
"status": "Failed",
"startTime": "2026-10-30T22:55:16.155391602Z",
"endTime": "2026-10-30T23:14:20.238434762Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/7beab753-b75d-4317-bea5-cd144607299f
correlation ID: faee6cbd-33e5-446e-9d93-e6c2de3830ab
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x27d5d703020>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xf9s7pl4h6wm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/6b933c0b-1e75-40d4-a318-42ac0d2371b2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/6b933c0b-1e75-40d4-a318-42ac0d2371b2",
"name": "6b933c0b-1e75-40d4-a318-42ac0d2371b2",
"status": "Failed",
"startTime": "2026-10-30T04:37:46.642962853Z",
"endTime": "2026-10-30T04:56:46.927194717Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/6b933c0b-1e75-40d4-a318-42ac0d2371b2
correlation ID: 998361e4-922f-4353-9b9a-53a3641fefe0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x27d5d703020>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xf9s7pl4h6wm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/6b933c0b-1e75-40d4-a318-42ac0d2371b2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/6b933c0b-1e75-40d4-a318-42ac0d2371b2",
"name": "6b933c0b-1e75-40d4-a318-42ac0d2371b2",
"status": "Failed",
"startTime": "2026-10-30T04:37:46.642962853Z",
"endTime": "2026-10-30T04:56:46.927194717Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/6b933c0b-1e75-40d4-a318-42ac0d2371b2
correlation ID: 998361e4-922f-4353-9b9a-53a3641fefe0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2535834a99c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rbmj59bntg5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1",
"name": "49137998-c82d-463c-9cd4-a8775024ebc1",
"status": "Failed",
"startTime": "2026-10-29T04:26:18.62948754Z",
"endTime": "2026-10-29T04:45:22.348417217Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
correlation ID: 40719e78-c8c1-4a49-a0a8-7e5f3f0cfa60
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2535834a99c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rbmj59bntg5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1",
"name": "49137998-c82d-463c-9cd4-a8775024ebc1",
"status": "Failed",
"startTime": "2026-10-29T04:26:18.62948754Z",
"endTime": "2026-10-29T04:45:22.348417217Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
correlation ID: 40719e78-c8c1-4a49-a0a8-7e5f3f0cfa60
client request ID: unavailable
occurredContributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | unknown | 2 | 3.85%2 of 52 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 1 · Nov 4: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||
Full failure examples (2){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-03T23:07:45Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-02T02:07:08Z"}Contributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.92%1 of 52 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 6 · Nov 4: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3f7c5959d280>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3f7c5959d280>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3dfb7093b40>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3dfb7093b40>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xa5400729d00>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:408]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xa5400729d00>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": net/http: TLS handshake timeout
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": net/http: TLS handshake timeout",
}Contributing tests (6)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.92%1 of 52 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 2 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | DEV | |||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sbvv9z24n27n/cluster-candidate-5-0-6t7kg7 should provision
Unexpected error:
<*fmt.wrapError | 0x6901d31c000>:
failed to create HCP cluster cluster-candidate-5-0-6t7kg7: failed waiting for cluster="cluster-candidate-5-0-6t7kg7" in resourcegroup="rg-candidate-5-0-sbvv9z24n27n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f2a45493-172e-4797-a34e-31bedea3247f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f2a45493-172e-4797-a34e-31bedea3247f",
"name": "f2a45493-172e-4797-a34e-31bedea3247f",
"status": "Failed",
"startTime": "2026-10-30T08:51:49.480086241Z",
"endTime": "2026-10-30T09:20:50.215342379Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0 is Partial (want Completed), started 23m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f2a45493-172e-4797-a34e-31bedea3247f
correlation ID: d0262992-0723-41e4-b22c-339b6f2c0aab
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sbvv9z24n27n/cluster-candidate-5-0-6t7kg7 should provision
Unexpected error:
<*fmt.wrapError | 0x6901d31c000>:
failed to create HCP cluster cluster-candidate-5-0-6t7kg7: failed waiting for cluster="cluster-candidate-5-0-6t7kg7" in resourcegroup="rg-candidate-5-0-sbvv9z24n27n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f2a45493-172e-4797-a34e-31bedea3247f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f2a45493-172e-4797-a34e-31bedea3247f",
"name": "f2a45493-172e-4797-a34e-31bedea3247f",
"status": "Failed",
"startTime": "2026-10-30T08:51:49.480086241Z",
"endTime": "2026-10-30T09:20:50.215342379Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0 is Partial (want Completed), started 23m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f2a45493-172e-4797-a34e-31bedea3247f
correlation ID: d0262992-0723-41e4-b22c-339b6f2c0aab
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-8qlvf6k84lvs/cluster-candidate-5-1-p98g5x should provision
Unexpected error:
<*fmt.wrapError | 0x3ef0d265c620>:
failed to create HCP cluster cluster-candidate-5-1-p98g5x: failed waiting for cluster="cluster-candidate-5-1-p98g5x" in resourcegroup="rg-candidate-5-1-8qlvf6k84lvs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c",
"name": "aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c",
"status": "Failed",
"startTime": "2026-10-30T08:52:03.286150211Z",
"endTime": "2026-10-30T09:21:10.230499944Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 22m31s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c
correlation ID: 36770878-c230-4f9d-a985-e8e04413f507
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-8qlvf6k84lvs/cluster-candidate-5-1-p98g5x should provision
Unexpected error:
<*fmt.wrapError | 0x3ef0d265c620>:
failed to create HCP cluster cluster-candidate-5-1-p98g5x: failed waiting for cluster="cluster-candidate-5-1-p98g5x" in resourcegroup="rg-candidate-5-1-8qlvf6k84lvs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c",
"name": "aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c",
"status": "Failed",
"startTime": "2026-10-30T08:52:03.286150211Z",
"endTime": "2026-10-30T09:21:10.230499944Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 22m31s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa6aaaf8-6dd2-463c-81fc-9946b5d8cf7c
correlation ID: 36770878-c230-4f9d-a985-e8e04413f507
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.92%1 of 52 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 1 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-dfv4zc" is viable
Unexpected error:
<*errors.joinError | 0x1851dc4f1470>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1851dc18a360>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1851dc18a340>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1851dc0e60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1851dbd57e20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1851dbd57de0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1851dbe77180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-dfv4zc" is viable
Unexpected error:
<*errors.joinError | 0x1851dc4f1470>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x1851dc18a360>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1851dc18a340>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1851dc0e60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1851dbd57e20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x1851dbd57de0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x1851dbe77180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 1.92%1 of 52 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsOct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 1 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-m6nrpn" is viable
Unexpected error:
<*errors.joinError | 0x2a7e709ee378>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2a7e705ff140>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2a7e705ff120>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2a7e7061a640>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-m6nrpn" is viable
Unexpected error:
<*errors.joinError | 0x2a7e709ee378>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2a7e705ff140>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2a7e705ff120>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2a7e7061a640>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 1 | 1.92%1 of 52 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 28: 1 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x243ce2483b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-r4jthp6b8c2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67",
"name": "4f6eb8f6-434d-4041-9e38-be33ab685c67",
"status": "Failed",
"startTime": "2026-10-29T13:48:20.576616469Z",
"endTime": "2026-10-29T14:07:20.745392823Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
correlation ID: 72c0068c-b7b6-49ea-9082-ad530d20a571
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x243ce2483b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-r4jthp6b8c2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67",
"name": "4f6eb8f6-434d-4041-9e38-be33ab685c67",
"status": "Failed",
"startTime": "2026-10-29T13:48:20.576616469Z",
"endTime": "2026-10-29T14:07:20.745392823Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
correlation ID: 72c0068c-b7b6-49ea-9082-ad530d20a571
client request ID: unavailableContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||