Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 94 | 57.45% (off track) | -37.55pp | +15.12pp | 84.78% (78/92) | -8.07pp | 69.23% (54/78) | +17.95pp |
| INT | 95% - All E2E job runs | 22 | 77.27% (off track) | -17.73pp | +8.38pp | n/a | n/a | 100.00% (17/17) | +3.12pp |
| STG | 95% - All E2E job runs | 22 | 77.27% (off track) | -17.73pp | +39.04pp | n/a | n/a | 80.95% (17/21) | +4.48pp |
| PROD | 95% - All E2E job runs | 91 | 36.26% (off track) | -58.74pp | -23.48pp | n/a | n/a | 36.67% (33/90) | -31.99pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | provision | 9 | 3.78%9 of 238 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 4: 0 · Nov 5: 4 · Nov 6: 4 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-09T12:56:42.099Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-09T12:56:43.631Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0?api-version=2021-04-01"
time=2026-11-09T12:56:45.820Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0?api-version=2021-04-01"
time=2026-11-09T12:56:50.272Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0?api-version=2021-04-01"
time=2026-11-09T12:56:58.880Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0?api-version=2021-04-01"
time=2026-11-09T12:57:15.482Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0?api-version=2021-04-01"
time=2026-11-09T12:57:50.004Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0?api-version=2021-04-01"
time=2026-11-09T12:57:50.004Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-09T12:57:51.041Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4251776-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0
time=2026-11-09T13:00:21.512Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/76fbd0bfa4873f5e518013577b8ea916a3ffa3537d8effdd972f20e89048d4c0/operationStatuses/08584073778149479288\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 7ff02092-b222-4c72-af33-dbc59d6681c7\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/cluster-43ztkdrarzwh4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 7ff02092-b222-4c72-af33-dbc59d6681c7\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j4251776-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j4251776-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-09T13:02:21.513Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-09T13:02:22.282Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780?api-version=2021-04-01"
time=2026-11-09T13:02:24.442Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780?api-version=2021-04-01"
time=2026-11-09T13:02:28.699Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780?api-version=2021-04-01"
time=2026-11-09T13:02:36.876Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780?api-version=2021-04-01"
time=2026-11-09T13:02:54.110Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780?api-version=2021-04-01"
time=2026-11-09T13:03:26.665Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780?api-version=2021-04-01"
time=2026-11-09T13:03:26.665Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-09T13:03:27.768Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4251776-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780
time=2026-11-09T13:04:58.026Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4251776-svc/providers/Microsoft.Resources/deployments/54f90093ac681f369f5c39d8ceea29882d59977e9688f7daf3feb0bbb5ce5780/operationStatuses/08584073774783229404\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: dee26b4e-35a3-482b-8248-1103b71c966a\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-06T17:00:38.021Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-06T17:00:39.542Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc?api-version=2021-04-01"
time=2026-11-06T17:00:41.785Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc?api-version=2021-04-01"
time=2026-11-06T17:00:46.131Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc?api-version=2021-04-01"
time=2026-11-06T17:00:54.515Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc?api-version=2021-04-01"
time=2026-11-06T17:01:10.664Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc?api-version=2021-04-01"
time=2026-11-06T17:01:45.922Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc?api-version=2021-04-01"
time=2026-11-06T17:01:45.922Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-06T17:01:46.934Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9826176-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc
time=2026-11-06T17:04:17.334Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/5371f59949f027ecd0bfe9a430fd6c9238a97662947313b038e53d1199bbbfbc/operationStatuses/08584076223790691749\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 992399a6-c1d8-4fea-b81c-a5d3cd846cb8\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/cluster-a4gc4wieu6pyq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 992399a6-c1d8-4fea-b81c-a5d3cd846cb8\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j9826176-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j9826176-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-06T17:06:17.335Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-06T17:06:18.074Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb?api-version=2021-04-01"
time=2026-11-06T17:06:20.232Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb?api-version=2021-04-01"
time=2026-11-06T17:06:24.363Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb?api-version=2021-04-01"
time=2026-11-06T17:06:33.211Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb?api-version=2021-04-01"
time=2026-11-06T17:06:49.702Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb?api-version=2021-04-01"
time=2026-11-06T17:07:23.100Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb?api-version=2021-04-01"
time=2026-11-06T17:07:23.100Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-06T17:07:24.667Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9826176-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb
time=2026-11-06T17:08:54.892Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9826176-svc/providers/Microsoft.Resources/deployments/f520811fc7af344f1d10ad7370b31bc8c0ec930c947b9b8097d712ac709377fb/operationStatuses/08584076220418331704\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 1279ea17-623d-46d2-86c4-006d6131d847\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-06T14:38:37.876Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-06T14:38:40.395Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1?api-version=2021-04-01"
time=2026-11-06T14:38:42.605Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1?api-version=2021-04-01"
time=2026-11-06T14:38:49.847Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1?api-version=2021-04-01"
time=2026-11-06T14:38:58.209Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1?api-version=2021-04-01"
time=2026-11-06T14:39:15.693Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1?api-version=2021-04-01"
time=2026-11-06T14:39:50.790Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1?api-version=2021-04-01"
time=2026-11-06T14:39:50.790Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-06T14:39:51.854Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0674688-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1
time=2026-11-06T14:41:52.396Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/ad781fbf105523304c2dba8805840396d21be7cc047c43543c17eef6e6237dd1/operationStatuses/08584076308941921261\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: fffd760c-6d2c-4509-ba9c-eff1e8a58274\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/cluster-233tlczqoaxd4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: fffd760c-6d2c-4509-ba9c-eff1e8a58274\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0674688-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j0674688-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-11-06T14:43:52.398Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-11-06T14:43:53.186Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8?api-version=2021-04-01"
time=2026-11-06T14:43:55.351Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8?api-version=2021-04-01"
time=2026-11-06T14:43:59.640Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8?api-version=2021-04-01"
time=2026-11-06T14:44:08.254Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8?api-version=2021-04-01"
time=2026-11-06T14:44:24.489Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8?api-version=2021-04-01"
time=2026-11-06T14:44:56.894Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8?api-version=2021-04-01"
time=2026-11-06T14:44:56.894Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-11-06T14:44:58.282Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0674688-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8
time=2026-11-06T14:46:28.681Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0674688-svc/providers/Microsoft.Resources/deployments/46f03ff06841a13aac5ff88ff655474bb3bcacf0d12c8bce2fa5fef5b05c64e8/operationStatuses/08584076305879556309\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: c62ba6b7-bde6-47e4-bc7f-d10f026774ba\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | alert | 8 | 3.36%8 of 238 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsNov 4: 2 · Nov 5: 5 · Nov 6: 1 · Nov 7: 0 · Nov 8: 1 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 22 time(s) (4 unknown, 18 known) Firing 1: State: Resolved Started: 2026-11-09T15:56:57Z Ended: 2026-11-09T16:20:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1504896-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tvpo9rk456n4na78beg9clr9ekkqajl-cluster", pod="router-647664f767-wckl4", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2tvpo9rk456n4na78beg9clr9ekkqajl-cluster/router-647664f767-wckl4 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-09T15:56:57Z Ended: 2026-11-09T16:20:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1504896-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tvpojn2ujcl0sb6piubi9r1a6kc7evq-cluster", pod="router-5f699cc44c-xr9g4", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2tvpojn2ujcl0sb6piubi9r1a6kc7evq-cluster/router-5f699cc44c-xr9g4 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-09T15:56:57Z Ended: 2026-11-09T16:20:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1504896-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tvpoc4b44m55n30bsrob33lmedumba0-cluster", pod="router-7dd8f45bdd-sjftf", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2tvpoc4b44m55n30bsrob33lmedumba0-cluster/router-7dd8f45bdd-sjftf has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-09T15:56:57Z Ended: 2026-11-09T16:20:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1504896-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tvpod6cghcrcfpfndvhv9f61bp64noo-cluster", pod="router-db8787ff9-s6dg6", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2tvpod6cghcrcfpfndvhv9f61bp64noo-cluster/router-db8787ff9-s6dg6 has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-08T22:08:03Z Ended: 2026-11-08T22:15:03Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6353280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tv9u2iliu4cuv1sv8r3qsirrdmei0q9-cluster", pod="openshift-apiserver-6bc596b9c-4s97m", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tv9u2iliu4cuv1sv8r3qsirrdmei0q9-cluster/openshift-apiserver-6bc596b9c-4s97m has been in a non-ready state for longer than 5 minutes. alert fired 55 time(s) (23 unknown, 32 known) Firing 1: State: Resolved Started: 2026-11-06T20:25:05Z Ended: 2026-11-06T20:40:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudpa45os2i56c3d8t2kfjqu7m8i7r-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudpa45os2i56c3d8t2kfjqu7m8i7r-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-06T20:25:05Z Ended: 2026-11-06T20:44:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttue9p5rsh1d0h77u9kt2c0fvt5ik5l-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttue9p5rsh1d0h77u9kt2c0fvt5ik5l-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-06T20:25:20Z Ended: 2026-11-06T20:44:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudpqsj2qju48ho51r57eah57b5mpj-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudpqsj2qju48ho51r57eah57b5mpj-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-06T20:25:20Z Ended: 2026-11-06T20:44:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudp4rufmetr0gae5eqhq4ncua7bc5-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudp4rufmetr0gae5eqhq4ncua7bc5-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-11-06T20:26:28Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudpcm937vff3lfdkgshl8rriur62f-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudpcm937vff3lfdkgshl8rriur62f-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-11-06T20:26:28Z Ended: 2026-11-06T20:44:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttue5v09s1ipo7vf62ojm2nb7e8kcfo-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttue5v09s1ipo7vf62ojm2nb7e8kcfo-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-11-06T20:26:28Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudqo00239dq5k6rq2egeu6819h2l2-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudqo00239dq5k6rq2egeu6819h2l2-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-11-06T20:27:55Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttuej6838hc2c6434e5cnn4un4oh5ol-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttuej6838hc2c6434e5cnn4un4oh5ol-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-11-06T20:27:57Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudp9v07slq4orsq0o3gdciacae433-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudp9v07slq4orsq0o3gdciacae433-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-11-06T20:27:57Z Ended: 2026-11-06T20:44:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttueup5ltcnt3k5669rl2ldb3im8vs5-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttueup5ltcnt3k5669rl2ldb3im8vs5-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-11-06T20:27:57Z Ended: 2026-11-06T20:44:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttuevlkt35cn88un8c0vd2hpkds8ofu-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttuevlkt35cn88un8c0vd2hpkds8ofu-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-11-06T20:28:12Z Ended: 2026-11-06T20:44:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudqd3q0lkp3umegu61dhidqnj0qgd-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudqd3q0lkp3umegu61dhidqnj0qgd-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-11-06T20:28:13Z Ended: 2026-11-06T20:44:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttueftdp3vfs2238rqoa0900bm5ourt-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttueftdp3vfs2238rqoa0900bm5ourt-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-11-06T20:31:06Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudqe0ni03ahi9g4gc776b38v669dj-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudqe0ni03ahi9g4gc776b38v669dj-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-11-06T20:31:06Z Ended: 2026-11-06T20:44:44Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudphhc6c70hk2qiu7ecnkliq4beao-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudphhc6c70hk2qiu7ecnkliq4beao-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-11-06T20:31:06Z Ended: 2026-11-06T20:44:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudmm1da6kjk8tnc37jra0aa0qp12b-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudmm1da6kjk8tnc37jra0aa0qp12b-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-11-06T20:31:06Z Ended: 2026-11-06T20:44:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttuebp9ap3jn8789aamfrk3rbmd08ku-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttuebp9ap3jn8789aamfrk3rbmd08ku-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-11-06T20:31:06Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttue4c7o05sipc6gdubal1agit8d61m-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttue4c7o05sipc6gdubal1agit8d61m-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-11-06T20:31:16Z Ended: 2026-11-06T20:44:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudn3j71afhsm4r89g5lte0q2revpk-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudn3j71afhsm4r89g5lte0q2revpk-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-11-06T20:31:18Z Ended: 2026-11-06T20:44:46Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudqpl1e5kr3tmenc8rqtnii5m9j61-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudqpl1e5kr3tmenc8rqtnii5m9j61-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-11-06T20:31:18Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttue8s3jvhhbhs9ftkj7ch42buiigfu-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttue8s3jvhhbhs9ftkj7ch42buiigfu-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-11-06T20:31:18Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttudqujeb0ju3o2perluqj5ggpd75fn-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttudqujeb0ju3o2perluqj5ggpd75fn-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-11-06T20:33:27Z Ended: 2026-11-06T20:44:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2574208-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ttue49vr0v1uud3vbjh799hifa29sb7-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2ttue49vr0v1uud3vbjh799hifa29sb7-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | e2e | 6 | 2.52%6 of 238 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsNov 4: 1 · Nov 5: 1 · Nov 6: 3 · Nov 7: 1 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2c89aca367e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
err: <*errors.errorString | 0x2c89ad98cef0>{
s: "dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2c89aca367e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7t6ckgnbcx5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
err: <*errors.errorString | 0x2c89ad98cef0>{
s: "dev/westus3/ci01-j8236544-mgmt-2/ocm-arohcpci01-2tvc5eed242rm1ms9gvk6h5g1inn2b1q-cluster router=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003: expected matching summary after test start and within 8m; summaryUID=3561f0a3-bf9d-4e4f-9c8a-6bad592a631f node=aks-userswft1-11422526-vmss000003 pass=20aefd59-6731-444c-8a7f-c10baba27548 at=2026-11-09T00:30:59Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x4rkj27qqr8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x14d4e2f80860>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x4rkj27qqr8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j1023744-mgmt-2/ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster router=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003: expected matching summary after test start and within 8m; summaryUID=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003 pass=a57e0e44-478c-482f-aac3-a98b87f95ffb at=2026-11-07T02:58:03Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x4rkj27qqr8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j1023744-mgmt-2/ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster router=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003: expected matching summary after test start and within 8m; summaryUID=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003 pass=a57e0e44-478c-482f-aac3-a98b87f95ffb at=2026-11-07T02:58:03Z",
err: <*verifiers.pollObservationError | 0x14d4e2f80840>{
observation: <*errors.errorString | 0x14d4e490c6c0>{
s: "dev/westus3/ci01-j1023744-mgmt-2/ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster router=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003: expected matching summary after test start and within 8m; summaryUID=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003 pass=a57e0e44-478c-482f-aac3-a98b87f95ffb at=2026-11-07T02:58:03Z",
},
interruption: <*fmt.wrapError | 0x14d4e2f80820>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1023744-mgmt-2\"; let ns = \"ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster\"; let start = datetime(2026-11-07T02:41:46.0968587Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x14d4e1fe96d0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x14d4e2f80800>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1023744-mgmt-2\"; let ns = \"ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster\"; let start = datetime(2026-11-07T02:41:46.0968587Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x14d4e4b8b650>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x4rkj27qqr8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x14d4e2f80860>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x4rkj27qqr8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j1023744-mgmt-2/ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster router=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003: expected matching summary after test start and within 8m; summaryUID=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003 pass=a57e0e44-478c-482f-aac3-a98b87f95ffb at=2026-11-07T02:58:03Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x4rkj27qqr8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j1023744-mgmt-2/ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster router=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003: expected matching summary after test start and within 8m; summaryUID=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003 pass=a57e0e44-478c-482f-aac3-a98b87f95ffb at=2026-11-07T02:58:03Z",
err: <*verifiers.pollObservationError | 0x14d4e2f80840>{
observation: <*errors.errorString | 0x14d4e490c6c0>{
s: "dev/westus3/ci01-j1023744-mgmt-2/ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster router=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003: expected matching summary after test start and within 8m; summaryUID=b25c3c50-9b1d-4711-b47e-70f682089390 node=aks-userswft2-32966811-vmss000003 pass=a57e0e44-478c-482f-aac3-a98b87f95ffb at=2026-11-07T02:58:03Z",
},
interruption: <*fmt.wrapError | 0x14d4e2f80820>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1023744-mgmt-2\"; let ns = \"ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster\"; let start = datetime(2026-11-07T02:41:46.0968587Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x14d4e1fe96d0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x14d4e2f80800>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1023744-mgmt-2\"; let ns = \"ocm-arohcpci01-2tu449dauakerhvlgp4smkd6hvm3nm1f-cluster\"; let start = datetime(2026-11-07T02:41:46.0968587Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x14d4e4b8b650>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-llxpl442sfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1e5c8da4620>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-llxpl442sfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7307904-mgmt-1/ocm-arohcpci01-2ttor6pj8t4mgrgg4kd0j8hb5j0o0nca-cluster router=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003: expected matching summary after test start and within 8m; summaryUID=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003 pass=d693d977-371d-4338-bb8f-ff0fe091988d at=2026-11-06T14:07:29Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-llxpl442sfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7307904-mgmt-1/ocm-arohcpci01-2ttor6pj8t4mgrgg4kd0j8hb5j0o0nca-cluster router=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003: expected matching summary after test start and within 8m; summaryUID=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003 pass=d693d977-371d-4338-bb8f-ff0fe091988d at=2026-11-06T14:07:29Z",
err: <*errors.errorString | 0x1e5c7e441d0>{
s: "dev/canadacentral/ci01-j7307904-mgmt-1/ocm-arohcpci01-2ttor6pj8t4mgrgg4kd0j8hb5j0o0nca-cluster router=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003: expected matching summary after test start and within 8m; summaryUID=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003 pass=d693d977-371d-4338-bb8f-ff0fe091988d at=2026-11-06T14:07:29Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-llxpl442sfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1e5c8da4620>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-llxpl442sfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7307904-mgmt-1/ocm-arohcpci01-2ttor6pj8t4mgrgg4kd0j8hb5j0o0nca-cluster router=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003: expected matching summary after test start and within 8m; summaryUID=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003 pass=d693d977-371d-4338-bb8f-ff0fe091988d at=2026-11-06T14:07:29Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-llxpl442sfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j7307904-mgmt-1/ocm-arohcpci01-2ttor6pj8t4mgrgg4kd0j8hb5j0o0nca-cluster router=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003: expected matching summary after test start and within 8m; summaryUID=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003 pass=d693d977-371d-4338-bb8f-ff0fe091988d at=2026-11-06T14:07:29Z",
err: <*errors.errorString | 0x1e5c7e441d0>{
s: "dev/canadacentral/ci01-j7307904-mgmt-1/ocm-arohcpci01-2ttor6pj8t4mgrgg4kd0j8hb5j0o0nca-cluster router=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003: expected matching summary after test start and within 8m; summaryUID=af0ebaf1-1b76-4fc5-a92e-79980591ea29 node=aks-userswft2-36057746-vmss000003 pass=d693d977-371d-4338-bb8f-ff0fe091988d at=2026-11-06T14:07:29Z",
},
}
occurredContributing tests (1)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGridfull failure pattern: ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | provision | 6 | 2.52%6 of 238 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 3 · Nov 10: 3 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-10T17:54:45.009Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-10T17:54:46.393Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc?api-version=2021-04-01"
time=2026-11-10T17:54:48.503Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc?api-version=2021-04-01"
time=2026-11-10T17:54:52.989Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc?api-version=2021-04-01"
time=2026-11-10T17:55:01.200Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc?api-version=2021-04-01"
time=2026-11-10T17:55:18.656Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc?api-version=2021-04-01"
time=2026-11-10T17:55:52.997Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc?api-version=2021-04-01"
time=2026-11-10T17:55:52.997Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-10T17:55:53.931Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1772672%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc
time=2026-11-10T18:12:26.762Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/bd37cc468adb65a0fe88d4d6c85dce65f5a6ce8eaa61b8aa111c24f12c97b2cc/operationStatuses/08584072735321234768\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 4ebe5d17-2520-4e4b-9982-9bc5ae4318de\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1772672/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 4ebe5d17-2520-4e4b-9982-9bc5ae4318de\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1772672/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1772672\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-10T17:48:58.875Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-10T17:49:00.402Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df?api-version=2021-04-01"
time=2026-11-10T17:49:02.516Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df?api-version=2021-04-01"
time=2026-11-10T17:49:06.987Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df?api-version=2021-04-01"
time=2026-11-10T17:49:15.365Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df?api-version=2021-04-01"
time=2026-11-10T17:49:32.312Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df?api-version=2021-04-01"
time=2026-11-10T17:50:07.655Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df?api-version=2021-04-01"
time=2026-11-10T17:50:07.656Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-10T17:50:08.492Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8549376%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df
time=2026-11-10T18:05:41.583Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/bfd2402bce59f4d51edbae53ff1f7e7aa4c2148a2c4bd4094ae40a40c82664df/operationStatuses/08584072738773491479\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 17729a1a-22ae-4559-84d2-4ca7ab21764b\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8549376/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 17729a1a-22ae-4559-84d2-4ca7ab21764b\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8549376/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j8549376\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-10T17:48:10.478Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-10T17:48:11.973Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7?api-version=2021-04-01"
time=2026-11-10T17:48:14.227Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7?api-version=2021-04-01"
time=2026-11-10T17:48:18.742Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7?api-version=2021-04-01"
time=2026-11-10T17:48:27.511Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7?api-version=2021-04-01"
time=2026-11-10T17:48:44.998Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7?api-version=2021-04-01"
time=2026-11-10T17:49:17.560Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7?api-version=2021-04-01"
time=2026-11-10T17:49:17.560Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-10T17:49:18.309Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2096512%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7
time=2026-11-10T18:04:52.253Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/cdf9137ef7b3b8e891fcbbd7006f5306b0817ae0286864f901fb29120a621dd7/operationStatuses/08584072739276616986\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 54b25005-9d2d-4104-b3d2-289f3f4532d4\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2096512/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 54b25005-9d2d-4104-b3d2-289f3f4532d4\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2096512/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j2096512\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | e2e | 5 | 2.10%5 of 238 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsNov 4: 1 · Nov 5: 2 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 1 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xxx7x4"
Unexpected error:
<*errors.joinError | 0x95ff8ec8030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xxx7x4-5bkgs-f87vd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xxx7x4-5bkgs-f87vd": nodes "cluster-np-xxx7x4-5bkgs-f87vd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x95ff9382460>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xxx7x4-5bkgs-f87vd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xxx7x4-5bkgs-f87vd\": nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x95ff9382440>{
msg: "failed to label node cluster-np-xxx7x4-5bkgs-f87vd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xxx7x4-5bkgs-f87vd\": nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x95ff9382420>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xxx7x4-5bkgs-f87vd\": nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x95ff93a06e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xxx7x4-5bkgs-f87vd",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xxx7x4"
Unexpected error:
<*errors.joinError | 0x95ff8ec8030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xxx7x4-5bkgs-f87vd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xxx7x4-5bkgs-f87vd": nodes "cluster-np-xxx7x4-5bkgs-f87vd" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x95ff9382460>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xxx7x4-5bkgs-f87vd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xxx7x4-5bkgs-f87vd\": nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x95ff9382440>{
msg: "failed to label node cluster-np-xxx7x4-5bkgs-f87vd egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xxx7x4-5bkgs-f87vd\": nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x95ff9382420>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xxx7x4-5bkgs-f87vd\": nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x95ff93a06e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xxx7x4-5bkgs-f87vd\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xxx7x4-5bkgs-f87vd",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-4-23-swpwl4"
Unexpected error:
<*errors.joinError | 0x34ff1b302540>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-swpwl4-jx5fl-52dk7 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-swpwl4-jx5fl-52dk7": nodes "cluster-np-swpwl4-jx5fl-52dk7" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x34ff1989c2c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-swpwl4-jx5fl-52dk7 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-swpwl4-jx5fl-52dk7\": nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x34ff1989c2a0>{
msg: "failed to label node cluster-np-swpwl4-jx5fl-52dk7 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-swpwl4-jx5fl-52dk7\": nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x34ff1989c280>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-swpwl4-jx5fl-52dk7\": nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x34ff1988d0e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-swpwl4-jx5fl-52dk7",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-4-23-swpwl4"
Unexpected error:
<*errors.joinError | 0x34ff1b302540>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-swpwl4-jx5fl-52dk7 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-swpwl4-jx5fl-52dk7": nodes "cluster-np-swpwl4-jx5fl-52dk7" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x34ff1989c2c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-swpwl4-jx5fl-52dk7 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-swpwl4-jx5fl-52dk7\": nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x34ff1989c2a0>{
msg: "failed to label node cluster-np-swpwl4-jx5fl-52dk7 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-swpwl4-jx5fl-52dk7\": nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x34ff1989c280>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-swpwl4-jx5fl-52dk7\": nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x34ff1988d0e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-swpwl4-jx5fl-52dk7\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-swpwl4-jx5fl-52dk7",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-z9bw2j"
Unexpected error:
<*errors.joinError | 0x240760eb6510>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-z9bw2j-bpg48-5jwn9 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-z9bw2j-bpg48-5jwn9": nodes "cluster-np-z9bw2j-bpg48-5jwn9" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x240760343720>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-z9bw2j-bpg48-5jwn9 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-z9bw2j-bpg48-5jwn9\": nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x240760343700>{
msg: "failed to label node cluster-np-z9bw2j-bpg48-5jwn9 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-z9bw2j-bpg48-5jwn9\": nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2407603436e0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-z9bw2j-bpg48-5jwn9\": nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x240760c199a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-z9bw2j-bpg48-5jwn9",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-z9bw2j"
Unexpected error:
<*errors.joinError | 0x240760eb6510>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-z9bw2j-bpg48-5jwn9 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-z9bw2j-bpg48-5jwn9": nodes "cluster-np-z9bw2j-bpg48-5jwn9" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x240760343720>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-z9bw2j-bpg48-5jwn9 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-z9bw2j-bpg48-5jwn9\": nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x240760343700>{
msg: "failed to label node cluster-np-z9bw2j-bpg48-5jwn9 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-z9bw2j-bpg48-5jwn9\": nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2407603436e0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-z9bw2j-bpg48-5jwn9\": nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x240760c199a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-z9bw2j-bpg48-5jwn9\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-z9bw2j-bpg48-5jwn9",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 5 | 2.10%5 of 238 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 5 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 4 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-hf4ltnkhjqx9/cluster-candidate-5-0-r6b4vh should provision
Unexpected error:
<*fmt.wrapError | 0x1f01542f0460>:
failed to create HCP cluster cluster-candidate-5-0-r6b4vh: failed waiting for cluster="cluster-candidate-5-0-r6b4vh" in resourcegroup="rg-candidate-5-0-hf4ltnkhjqx9" to finish creating: GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77",
"name": "dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77",
"status": "Failed",
"startTime": "2026-11-09T15:43:42.434317825Z",
"endTime": "2026-11-09T16:12:51.677888569Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.1 is Partial (want Completed), started 23m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77
correlation ID: d7d54ef3-e6d1-4b29-aab2-227f59d68320
client request ID: d0059588-f0d5-4ba9-a690-2a85f3ee31ad
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-hf4ltnkhjqx9/cluster-candidate-5-0-r6b4vh should provision
Unexpected error:
<*fmt.wrapError | 0x1f01542f0460>:
failed to create HCP cluster cluster-candidate-5-0-r6b4vh: failed waiting for cluster="cluster-candidate-5-0-r6b4vh" in resourcegroup="rg-candidate-5-0-hf4ltnkhjqx9" to finish creating: GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77",
"name": "dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77",
"status": "Failed",
"startTime": "2026-11-09T15:43:42.434317825Z",
"endTime": "2026-11-09T16:12:51.677888569Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.1 is Partial (want Completed), started 23m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae1d9cd-a6b8-4c11-93d7-0f5b7c0daf77
correlation ID: d7d54ef3-e6d1-4b29-aab2-227f59d68320
client request ID: d0059588-f0d5-4ba9-a690-2a85f3ee31ad
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-8fn24lqct5gz/cluster-candidate-4-22-r95mt2 should provision
Unexpected error:
<*fmt.wrapError | 0x6a72010e2a0>:
failed to create HCP cluster cluster-candidate-4-22-r95mt2: failed waiting for cluster="cluster-candidate-4-22-r95mt2" in resourcegroup="rg-candidate-4-22-8fn24lqct5gz" to finish creating: GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9d57001-7202-4f40-b781-9f2f8508e376
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9d57001-7202-4f40-b781-9f2f8508e376",
"name": "c9d57001-7202-4f40-b781-9f2f8508e376",
"status": "Failed",
"startTime": "2026-11-09T15:43:41.555049379Z",
"endTime": "2026-11-09T16:12:41.473179052Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9d57001-7202-4f40-b781-9f2f8508e376
correlation ID: 000ffdc6-6c95-4edf-85df-2831102fefa1
client request ID: 6fa9c0d9-4855-44f7-9d8a-627f13086fee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-8fn24lqct5gz/cluster-candidate-4-22-r95mt2 should provision
Unexpected error:
<*fmt.wrapError | 0x6a72010e2a0>:
failed to create HCP cluster cluster-candidate-4-22-r95mt2: failed waiting for cluster="cluster-candidate-4-22-r95mt2" in resourcegroup="rg-candidate-4-22-8fn24lqct5gz" to finish creating: GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9d57001-7202-4f40-b781-9f2f8508e376
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9d57001-7202-4f40-b781-9f2f8508e376",
"name": "c9d57001-7202-4f40-b781-9f2f8508e376",
"status": "Failed",
"startTime": "2026-11-09T15:43:41.555049379Z",
"endTime": "2026-11-09T16:12:41.473179052Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9d57001-7202-4f40-b781-9f2f8508e376
correlation ID: 000ffdc6-6c95-4edf-85df-2831102fefa1
client request ID: 6fa9c0d9-4855-44f7-9d8a-627f13086fee
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:107]: failed to create fully private HCP cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x2f1bf2fdafc0>:
failed to create HCP cluster fully-private: failed waiting for cluster="fully-private" in resourcegroup="fully-private-h6s2gqwzc9rg" to finish creating: GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/58553889-517b-4c1d-b334-95ead80091f7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/58553889-517b-4c1d-b334-95ead80091f7",
"name": "58553889-517b-4c1d-b334-95ead80091f7",
"status": "Failed",
"startTime": "2026-11-09T15:44:14.289877945Z",
"endTime": "2026-11-09T16:13:21.914454363Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 23m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/58553889-517b-4c1d-b334-95ead80091f7
correlation ID: b3f095b9-eba0-4e14-a987-8e8c22c18bb5
client request ID: 5f64e85c-e8ff-45dd-a58c-4ce96c02ea41
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:107]: failed to create fully private HCP cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x2f1bf2fdafc0>:
failed to create HCP cluster fully-private: failed waiting for cluster="fully-private" in resourcegroup="fully-private-h6s2gqwzc9rg" to finish creating: GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/58553889-517b-4c1d-b334-95ead80091f7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/58553889-517b-4c1d-b334-95ead80091f7",
"name": "58553889-517b-4c1d-b334-95ead80091f7",
"status": "Failed",
"startTime": "2026-11-09T15:44:14.289877945Z",
"endTime": "2026-11-09T16:13:21.914454363Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 23m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1504896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/58553889-517b-4c1d-b334-95ead80091f7
correlation ID: b3f095b9-eba0-4e14-a987-8e8c22c18bb5
client request ID: 5f64e85c-e8ff-45dd-a58c-4ce96c02ea41
occurredContributing tests (8)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | unknown | 5 | 2.10%5 of 238 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 4: 2 · Nov 5: 5 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | INT | ||||||||||||||||||||||||||||||||||||
Full failure examples (3){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T12:34:57Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T12:37:40Z"}{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T12:32:07Z"}Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...full failure pattern: ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDBSignal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | provision | 4 | 1.68%4 of 238 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsNov 4: 2 · Nov 5: 1 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 1 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-10T11:16:22.539Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-10T11:16:24.181Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:26.335Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:30.679Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:39.545Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:16:55.947Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:17:31.257Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10?api-version=2021-04-01"
time=2026-11-10T11:17:31.257Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-10T11:17:31.890Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7760256%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10
time=2026-11-10T11:24:03.481Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/c86eb72dce948b4410c5375af5f35ec0fc9521fed642e85ede55cad109c39f10/operationStatuses/08584072974339535323\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e9565745-82ab-4caf-abba-d4474320716f\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7760256/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e9565745-82ab-4caf-abba-d4474320716f\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7760256/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j7760256\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: e6bf8d0e-46dd-459a-9496-7274d225014a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 03242ad1-50e6-4647-a472-8bcee3bf6ab9, Request URI: /apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/285f097a-abb1-4c88-9d14-c42518f81a10/partitions/0f0d6a3c-7223-4cbb-a460-003329156d27/replicas/134383683272148038s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-10T11:23:08.2315094Z, RequestEndTime: 2026-11-10T11:23:13.2160337Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:20.7122803Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.346,\\\\\\\"memory\\\\\\\":23804772.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0467,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:30.7282997Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.579,\\\\\\\"memory\\\\\\\":23813272.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.096,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:40.7301913Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.109,\\\\\\\"memory\\\\\\\":23816912.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0604,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:22:50.7460385Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.814,\\\\\\\"memory\\\\\\\":23813536.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.065,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:23:00.7617917Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.755,\\\\\\\"memory\\\\\\\":23802572.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0864,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-10T11:23:10.7628025Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.348,\\\\\\\"memory\\\\\\\":23788572.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0913,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":94}]}\\\\r\\\\nRequestStart: 2026-11-10T11:23:08.2315094Z; ResponseTime: 2026-11-10T11:23:13.2160337Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be149.documents.azure.com:11300/apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/285f097a-abb1-4c88-9d14-c42518f81a10/partitions/0f0d6a3c-7223-4cbb-a460-003329156d27/replicas/134383683272148038s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 4987.977, ActivityId: 03242ad1-50e6-4647-a472-8bcee3bf6ab9, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11300 | status: Connected | lkt: 11/10/2026 11:23:13 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2315094Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0028},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2315122Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0085},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2315207Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0911},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:08.2316118Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 4988.5594},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:13.2201712Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1192},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-10T11:23:13.2202904Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-10T11:20:55.0207558Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-10T11:20:55.0207558Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-10T11:21:00.2239532Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-09T21:54:28.988Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-09T21:54:30.623Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86?api-version=2021-04-01"
time=2026-11-09T21:54:33.028Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86?api-version=2021-04-01"
time=2026-11-09T21:54:37.324Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86?api-version=2021-04-01"
time=2026-11-09T21:54:45.933Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86?api-version=2021-04-01"
time=2026-11-09T21:55:02.670Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86?api-version=2021-04-01"
time=2026-11-09T21:55:37.181Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86?api-version=2021-04-01"
time=2026-11-09T21:55:37.181Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-09T21:55:38.553Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8187136%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86
time=2026-11-09T22:02:10.140Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/f357d5eacbc4fd6763c007ca1654765e68187f230f2fe49c8583f27259d77a86/operationStatuses/08584073455477316041\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 41095ac4-f325-4d66-96ff-1a966fe7774a\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8187136/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 41095ac4-f325-4d66-96ff-1a966fe7774a\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8187136/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j8187136\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 9bc80b19-9dd2-489a-a235-acab775ac91a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: ffe9e57a-a304-4709-b212-3c07e6a51cde, Request URI: /apps/59f2fa34-af0a-4c1b-9a03-14565c0ad612/services/089226f7-0495-478c-97cb-707f648e3670/partitions/4618a2f0-8c29-4a25-8d05-2b3e1416524e/replicas/134352771115111780s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-09T22:01:23.9412668Z, RequestEndTime: 2026-11-09T22:01:29.9275388Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-09T22:00:37.9290575Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.443,\\\\\\\"memory\\\\\\\":24914588.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0922,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":55},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-09T22:00:47.9345863Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.856,\\\\\\\"memory\\\\\\\":24863136.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1329,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":55},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-09T22:00:57.9458984Z\\\\\\\",\\\\\\\"cpu\\\\\\\":6.669,\\\\\\\"memory\\\\\\\":24869168.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.109,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":55},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-09T22:01:07.9515792Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.656,\\\\\\\"memory\\\\\\\":24860012.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0656,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":60},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-09T22:01:17.9522718Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.406,\\\\\\\"memory\\\\\\\":24869108.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1549,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":60},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-09T22:01:27.9586809Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.501,\\\\\\\"memory\\\\\\\":24881220.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0569,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":64}]}\\\\r\\\\nRequestStart: 2026-11-09T22:01:23.9412668Z; ResponseTime: 2026-11-09T22:01:29.9275388Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-westus3-be204.documents.azure.com:11613/apps/59f2fa34-af0a-4c1b-9a03-14565c0ad612/services/089226f7-0495-478c-97cb-707f648e3670/partitions/4618a2f0-8c29-4a25-8d05-2b3e1416524e/replicas/134352771115111780s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5982.062, ActivityId: ffe9e57a-a304-4709-b212-3c07e6a51cde, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11613 | status: Connected | lkt: 11/9/2026 10:01:29 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-09T22:01:23.9412668Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0028},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-09T22:01:23.9412696Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.008},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-09T22:01:23.9412776Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1054},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-09T22:01:23.9413830Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5982.9145},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-09T22:01:29.9242975Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1707},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-09T22:01:29.9244682Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-09T21:59:10.6864168Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-09T21:59:10.6864168Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-09T21:59:15.9208723Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":873,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-06T10:28:19.342Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-06T10:28:21.008Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1?api-version=2021-04-01"
time=2026-11-06T10:28:23.181Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1?api-version=2021-04-01"
time=2026-11-06T10:28:27.596Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1?api-version=2021-04-01"
time=2026-11-06T10:28:36.093Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1?api-version=2021-04-01"
time=2026-11-06T10:28:52.210Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1?api-version=2021-04-01"
time=2026-11-06T10:29:25.444Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1?api-version=2021-04-01"
time=2026-11-06T10:29:25.444Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-06T10:29:26.181Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9041536%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1
time=2026-11-06T10:35:58.040Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/29194c849dc770f2b5da7f3da7acc96ffa067c149f123585a07db2b741f944f1/operationStatuses/08584076459197706081\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 0871d653-b940-4214-9886-1cea605a9b28\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9041536/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 0871d653-b940-4214-9886-1cea605a9b28\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9041536/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j9041536\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: a94158c2-6406-49d3-ac57-0bb38c3c7dfa, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 8a3aa909-7e45-4596-a12f-f66da8a50892, Request URI: /apps/4f2607c3-d28a-4c86-bf29-0e8a305cd7d1/services/6f3ab4aa-24b8-47ca-b6f6-bd0dd3ba5bd9/partitions/f82a9cef-efde-41c0-8e93-42b3af97afc1/replicas/134377649296592197p, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-06T10:35:14.7252234Z, RequestEndTime: 2026-11-06T10:35:20.1159483Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-06T10:34:27.4144125Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.172,\\\\\\\"memory\\\\\\\":24955448.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0931,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":306},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-06T10:34:37.4301278Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.619,\\\\\\\"memory\\\\\\\":24960976.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1051,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":309},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-06T10:34:47.4457967Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.618,\\\\\\\"memory\\\\\\\":24968280.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0802,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":311},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-06T10:34:57.4615118Z\\\\\\\",\\\\\\\"cpu\\\\\\\":5.616,\\\\\\\"memory\\\\\\\":24885412.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.046,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":311},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-06T10:35:07.4751654Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.360,\\\\\\\"memory\\\\\\\":24891456.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0931,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":311},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-06T10:35:17.4909058Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.638,\\\\\\\"memory\\\\\\\":24910184.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.072,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":311}]}\\\\r\\\\nRequestStart: 2026-11-06T10:35:14.7252234Z; ResponseTime: 2026-11-06T10:35:20.1159483Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be149.documents.azure.com:11600/apps/4f2607c3-d28a-4c86-bf29-0e8a305cd7d1/services/6f3ab4aa-24b8-47ca-b6f6-bd0dd3ba5bd9/partitions/f82a9cef-efde-41c0-8e93-42b3af97afc1/replicas/134377649296592197p, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5393.038, ActivityId: 8a3aa909-7e45-4596-a12f-f66da8a50892, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11600 | status: Connected | lkt: 11/6/2026 10:35:20 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-06T10:35:14.7252234Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0067},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-06T10:35:14.7252301Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0214},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-06T10:35:14.7252515Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.3447},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-06T10:35:14.7255962Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5393.9158},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-06T10:35:20.1195120Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1915},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-06T10:35:20.1197035Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-06T10:33:01.1273680Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-06T10:33:01.1273680Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-06T10:33:05.8305148Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":871,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 4 | 1.68%4 of 238 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 4: 1 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 2 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htzcfl5gn6zs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1f728e1f53a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htzcfl5gn6zs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1713664-mgmt-2/ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster router=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004: expected matching summary after test start and within 8m; summaryUID=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004 pass=8287012a-c202-4669-a10a-47011d6f9a27 at=2026-11-08T21:59:18Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htzcfl5gn6zs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1713664-mgmt-2/ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster router=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004: expected matching summary after test start and within 8m; summaryUID=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004 pass=8287012a-c202-4669-a10a-47011d6f9a27 at=2026-11-08T21:59:18Z",
err: <*verifiers.pollObservationError | 0x1f728e1f5380>{
observation: <*errors.errorString | 0x1f728e1e2090>{
s: "dev/centralus/ci01-j1713664-mgmt-2/ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster router=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004: expected matching summary after test start and within 8m; summaryUID=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004 pass=8287012a-c202-4669-a10a-47011d6f9a27 at=2026-11-08T21:59:18Z",
},
interruption: <*fmt.wrapError | 0x1f728e1f5360>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1713664-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster\"; let start = datetime(2026-11-08T21:43:01.5456122Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1f728d742780>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1f728e1f5340>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1713664-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster\"; let start = datetime(2026-11-08T21:43:01.5456122Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1f728f687290>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htzcfl5gn6zs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1f728e1f53a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htzcfl5gn6zs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1713664-mgmt-2/ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster router=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004: expected matching summary after test start and within 8m; summaryUID=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004 pass=8287012a-c202-4669-a10a-47011d6f9a27 at=2026-11-08T21:59:18Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htzcfl5gn6zs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j1713664-mgmt-2/ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster router=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004: expected matching summary after test start and within 8m; summaryUID=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004 pass=8287012a-c202-4669-a10a-47011d6f9a27 at=2026-11-08T21:59:18Z",
err: <*verifiers.pollObservationError | 0x1f728e1f5380>{
observation: <*errors.errorString | 0x1f728e1e2090>{
s: "dev/centralus/ci01-j1713664-mgmt-2/ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster router=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004: expected matching summary after test start and within 8m; summaryUID=11417d3f-34f9-4cf2-9cdc-b457d70e2f0e node=aks-userswft3-33752756-vmss000004 pass=8287012a-c202-4669-a10a-47011d6f9a27 at=2026-11-08T21:59:18Z",
},
interruption: <*fmt.wrapError | 0x1f728e1f5360>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1713664-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster\"; let start = datetime(2026-11-08T21:43:01.5456122Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1f728d742780>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1f728e1f5340>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j1713664-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv9u54bvliqtcf8uakjvv5k078tk04g-cluster\"; let start = datetime(2026-11-08T21:43:01.5456122Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1f728f687290>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fcm59nkk9nls/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xd2046582340>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fcm59nkk9nls/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j8934784-mgmt-2/ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster router=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001: expected matching summary after test start and within 8m; summaryUID=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001 pass=22373cf9-ebef-4ce4-997b-461f85a3e80c at=2026-11-08T14:17:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fcm59nkk9nls/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j8934784-mgmt-2/ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster router=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001: expected matching summary after test start and within 8m; summaryUID=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001 pass=22373cf9-ebef-4ce4-997b-461f85a3e80c at=2026-11-08T14:17:59Z",
err: <*verifiers.pollObservationError | 0xd2046582320>{
observation: <*errors.errorString | 0xd2045249520>{
s: "dev/centralus/ci01-j8934784-mgmt-2/ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster router=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001: expected matching summary after test start and within 8m; summaryUID=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001 pass=22373cf9-ebef-4ce4-997b-461f85a3e80c at=2026-11-08T14:17:59Z",
},
interruption: <*fmt.wrapError | 0xd2046582300>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8934784-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster\"; let start = datetime(2026-11-08T14:00:33.8075577Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0xd20456fe6e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0xd20465822e0>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8934784-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster\"; let start = datetime(2026-11-08T14:00:33.8075577Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0xd20475d0420>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fcm59nkk9nls/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xd2046582340>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fcm59nkk9nls/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j8934784-mgmt-2/ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster router=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001: expected matching summary after test start and within 8m; summaryUID=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001 pass=22373cf9-ebef-4ce4-997b-461f85a3e80c at=2026-11-08T14:17:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fcm59nkk9nls/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j8934784-mgmt-2/ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster router=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001: expected matching summary after test start and within 8m; summaryUID=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001 pass=22373cf9-ebef-4ce4-997b-461f85a3e80c at=2026-11-08T14:17:59Z",
err: <*verifiers.pollObservationError | 0xd2046582320>{
observation: <*errors.errorString | 0xd2045249520>{
s: "dev/centralus/ci01-j8934784-mgmt-2/ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster router=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001: expected matching summary after test start and within 8m; summaryUID=275b3392-b95e-4d5b-a0ed-c645db1f6c42 node=aks-userswft3-27661241-vmss000001 pass=22373cf9-ebef-4ce4-997b-461f85a3e80c at=2026-11-08T14:17:59Z",
},
interruption: <*fmt.wrapError | 0xd2046582300>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8934784-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster\"; let start = datetime(2026-11-08T14:00:33.8075577Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0xd20456fe6e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0xd20465822e0>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8934784-mgmt-2\"; let ns = \"ocm-arohcpci01-2tv359jef4mpd5loreaipku10qm23iu0-cluster\"; let start = datetime(2026-11-08T14:00:33.8075577Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0xd20475d0420>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-pxcdsjqks5v6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3fd615312000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-pxcdsjqks5v6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/westus3/ci01-j4595328-mgmt-2/ocm-arohcpci01-2ttngn2fjcuvtd68ocra97scsq8mvaab-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-pxcdsjqks5v6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/westus3/ci01-j4595328-mgmt-2/ocm-arohcpci01-2ttngn2fjcuvtd68ocra97scsq8mvaab-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x3fd614e09ea0>{
s: "dev/westus3/ci01-j4595328-mgmt-2/ocm-arohcpci01-2ttngn2fjcuvtd68ocra97scsq8mvaab-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-pxcdsjqks5v6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3fd615312000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-pxcdsjqks5v6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/westus3/ci01-j4595328-mgmt-2/ocm-arohcpci01-2ttngn2fjcuvtd68ocra97scsq8mvaab-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-pxcdsjqks5v6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/westus3/ci01-j4595328-mgmt-2/ocm-arohcpci01-2ttngn2fjcuvtd68ocra97scsq8mvaab-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x3fd614e09ea0>{
s: "dev/westus3/ci01-j4595328-mgmt-2/ocm-arohcpci01-2ttngn2fjcuvtd68ocra97scsq8mvaab-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredContributing tests (1)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 4 | 1.68%4 of 238 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-sxdzxc"
Unexpected error:
<*fmt.wrapError | 0x7b34d13e720>:
DNS for route host agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7b34cd06c80>{
msg: "DNS for agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-sxdzxc"
Unexpected error:
<*fmt.wrapError | 0x7b34d13e720>:
DNS for route host agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7b34cd06c80>{
msg: "DNS for agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fcfr8.apps.aro.cluster.6tgwl.j3977344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-n6gsxq"
Unexpected error:
<*fmt.wrapError | 0x3e280d3fc8a0>:
DNS for route host agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3e280e3820e0>{
msg: "DNS for agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-n6gsxq"
Unexpected error:
<*fmt.wrapError | 0x3e280d3fc8a0>:
DNS for route host agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3e280e3820e0>{
msg: "DNS for agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mvht7.apps.aro.cluster.2zmkd.j0884608.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-7htxrc"
Unexpected error:
<*fmt.wrapError | 0x235267c24a80>:
DNS for route host agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x235267b88760>{
msg: "DNS for agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:249]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-7htxrc"
Unexpected error:
<*fmt.wrapError | 0x235267c24a80>:
DNS for route host agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x235267b88760>{
msg: "DNS for agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-l9jv5.apps.aro.cluster.9l6c6.j1634304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (3)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | alert | 4 | 1.68%4 of 238 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 3 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-11-07T00:12:56Z Ended: 2026-11-07T00:20:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8837888-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-5c4b485795-fx7hp", region="centralus", severity="warning" Description: Pod clusters-service/clusters-service-5c4b485795-fx7hp has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-07T00:12:56Z Ended: 2026-11-07T00:20:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8837888-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-5c4b485795-xn6gj", region="centralus", severity="warning" Description: Pod clusters-service/clusters-service-5c4b485795-xn6gj has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-06T13:45:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0250880-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-68dc47d4b9-dvtv9", region="centralus", severity="warning" Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-68dc47d4b9-dvtv9 has been in a non-ready state for longer than 5 minutes. alert fired 3 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-06T08:41:38Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6964992-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-d7669cb4-q7ngb", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-d7669cb4-q7ngb has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-06T08:41:38Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6964992-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-d7669cb4-48tf2", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-d7669cb4-48tf2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-11-06T08:42:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6964992-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-d7669cb4-ztf58", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-d7669cb4-ztf58 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 3 | 1.26%3 of 238 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsNov 4: 1 · Nov 5: 1 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x31589f922000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
err: <*errors.errorString | 0x31589ee10950>{
s: "dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x31589f922000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-g5zb6x5s4z5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
err: <*errors.errorString | 0x31589ee10950>{
s: "dev/centralus/ci01-j5777408-mgmt-2/ocm-arohcpci01-2tvpg8s169hijh28ajlhl8bdchbtnlqf-cluster router=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002: expected matching summary after test start and within 8m; summaryUID=e53bc550-a240-4615-a789-da036d51a38c node=aks-userswft2-14170224-vmss000002 pass=67b8dac0-c47b-4be4-af46-ec6140b4db75 at=2026-11-09T15:39:33Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dlfcvg5xmpkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1e7be204e020>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dlfcvg5xmpkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9363712-mgmt-1/ocm-arohcpci01-2ttu83ont82m7n1rhkcvnva0v54voegp-cluster router=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002: expected matching summary after test start and within 8m; summaryUID=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002 pass=22816248-356f-49dc-877c-9564e54118ca at=2026-11-06T20:14:58Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dlfcvg5xmpkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9363712-mgmt-1/ocm-arohcpci01-2ttu83ont82m7n1rhkcvnva0v54voegp-cluster router=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002: expected matching summary after test start and within 8m; summaryUID=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002 pass=22816248-356f-49dc-877c-9564e54118ca at=2026-11-06T20:14:58Z",
err: <*errors.errorString | 0x1e7be2d018e0>{
s: "dev/centralus/ci01-j9363712-mgmt-1/ocm-arohcpci01-2ttu83ont82m7n1rhkcvnva0v54voegp-cluster router=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002: expected matching summary after test start and within 8m; summaryUID=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002 pass=22816248-356f-49dc-877c-9564e54118ca at=2026-11-06T20:14:58Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dlfcvg5xmpkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1e7be204e020>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dlfcvg5xmpkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9363712-mgmt-1/ocm-arohcpci01-2ttu83ont82m7n1rhkcvnva0v54voegp-cluster router=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002: expected matching summary after test start and within 8m; summaryUID=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002 pass=22816248-356f-49dc-877c-9564e54118ca at=2026-11-06T20:14:58Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dlfcvg5xmpkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9363712-mgmt-1/ocm-arohcpci01-2ttu83ont82m7n1rhkcvnva0v54voegp-cluster router=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002: expected matching summary after test start and within 8m; summaryUID=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002 pass=22816248-356f-49dc-877c-9564e54118ca at=2026-11-06T20:14:58Z",
err: <*errors.errorString | 0x1e7be2d018e0>{
s: "dev/centralus/ci01-j9363712-mgmt-1/ocm-arohcpci01-2ttu83ont82m7n1rhkcvnva0v54voegp-cluster router=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002: expected matching summary after test start and within 8m; summaryUID=e486ea7b-03e5-4267-9a2c-bc6dcc9436b9 node=aks-userswft1-63551506-vmss000002 pass=22816248-356f-49dc-877c-9564e54118ca at=2026-11-06T20:14:58Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x551cc2ce440>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x551cbff6a70>{
s: "dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x551cc2ce440>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-r5mx54j9dbkk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x551cbff6a70>{
s: "dev/westus3/ci01-j1160960-mgmt-2/ocm-arohcpci01-2tt1fq1b1safcbc1bpt9anam5lqs2gqi-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.26%3 of 238 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 4: 1 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tgl4ks7cjj85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x533a8565820>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tgl4ks7cjj85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tgl4ks7cjj85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
err: <*verifiers.pollObservationError | 0x533a8565800>{
observation: <*errors.errorString | 0x533a979f300>{
s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
},
interruption: <*fmt.wrapError | 0x533a8565740>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4225408-mgmt-1\"; let ns = \"ocm-arohcpci01-2tvqm4eekn1nle4uofqvbb99m2qkvbvf-cluster\"; let start = datetime(2026-11-09T16:46:07.8659517Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x533a8748370>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x533a8565720>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4225408-mgmt-1\"; let ns = \"ocm-arohcpci01-2tvqm4eekn1nle4uofqvbb99m2qkvbvf-cluster\"; let start = datetime(2026-11-09T16:46:07.8659517Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x533aab013b0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tgl4ks7cjj85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x533a8565820>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tgl4ks7cjj85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tgl4ks7cjj85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
err: <*verifiers.pollObservationError | 0x533a8565800>{
observation: <*errors.errorString | 0x533a979f300>{
s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
},
interruption: <*fmt.wrapError | 0x533a8565740>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4225408-mgmt-1\"; let ns = \"ocm-arohcpci01-2tvqm4eekn1nle4uofqvbb99m2qkvbvf-cluster\"; let start = datetime(2026-11-09T16:46:07.8659517Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x533a8748370>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x533a8565720>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j4225408-mgmt-1\"; let ns = \"ocm-arohcpci01-2tvqm4eekn1nle4uofqvbb99m2qkvbvf-cluster\"; let start = datetime(2026-11-09T16:46:07.8659517Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x533aab013b0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x46n98lwzk77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x31ebb26da780>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x46n98lwzk77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x46n98lwzk77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
err: <*verifiers.pollObservationError | 0x31ebb26da760>{
observation: <*errors.errorString | 0x31ebb276b2e0>{
s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
},
interruption: <*fmt.wrapError | 0x31ebb26da740>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j1792128-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttp6t2dmiocush8hsjo94jgce6n5idc-cluster\"; let start = datetime(2026-11-06T14:15:43.561424Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x31ebb168ea50>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x31ebb26da720>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j1792128-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttp6t2dmiocush8hsjo94jgce6n5idc-cluster\"; let start = datetime(2026-11-06T14:15:43.561424Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x31ebb2ffc2a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x46n98lwzk77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x31ebb26da780>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x46n98lwzk77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x46n98lwzk77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
err: <*verifiers.pollObservationError | 0x31ebb26da760>{
observation: <*errors.errorString | 0x31ebb276b2e0>{
s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
},
interruption: <*fmt.wrapError | 0x31ebb26da740>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j1792128-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttp6t2dmiocush8hsjo94jgce6n5idc-cluster\"; let start = datetime(2026-11-06T14:15:43.561424Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x31ebb168ea50>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x31ebb26da720>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j1792128-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttp6t2dmiocush8hsjo94jgce6n5idc-cluster\"; let start = datetime(2026-11-06T14:15:43.561424Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x31ebb2ffc2a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dkhftxl7gvwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1e6b08aae80>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dkhftxl7gvwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dkhftxl7gvwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
err: <*verifiers.pollObservationError | 0x1e6b08aae40>{
observation: <*errors.errorString | 0x1e6b22205f0>{
s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
},
interruption: <*fmt.wrapError | 0x1e6b08aade0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2875264-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttne46ahs9blap0lfnkeso6nanrbe0r-cluster\"; let start = datetime(2026-11-06T12:15:20.3131762Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1e6afe7e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1e6b08aadc0>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2875264-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttne46ahs9blap0lfnkeso6nanrbe0r-cluster\"; let start = datetime(2026-11-06T12:15:20.3131762Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1e6b1a74cc0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dkhftxl7gvwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1e6b08aae80>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dkhftxl7gvwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dkhftxl7gvwj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
err: <*verifiers.pollObservationError | 0x1e6b08aae40>{
observation: <*errors.errorString | 0x1e6b22205f0>{
s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
},
interruption: <*fmt.wrapError | 0x1e6b08aade0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2875264-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttne46ahs9blap0lfnkeso6nanrbe0r-cluster\"; let start = datetime(2026-11-06T12:15:20.3131762Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1e6afe7e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1e6b08aadc0>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2875264-mgmt-1\"; let ns = \"ocm-arohcpci01-2ttne46ahs9blap0lfnkeso6nanrbe0r-cluster\"; let start = datetime(2026-11-06T12:15:20.3131762Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1e6b1a74cc0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerRetryHotLoop firedfull failure pattern: alert [svc] BackendControllerRetryHotLoop fired Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | alert | 3 | 1.26%3 of 238 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-10T16:03:59Z Ended: 2026-11-10T16:09:59Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j4690432-svc", component="backend", name="azuredeploymentstackdesires", region="canadacentral", severity="warning" Description: Backend controller workqueue azuredeploymentstackdesires has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-06T23:24:22Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j3059200-svc", component="backend", name="clusterservicematchingclusters", region="westus3", severity="warning" Description: Backend controller workqueue clusterservicematchingclusters has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-06T18:20:39Z Ended: 2026-11-06T18:26:36Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j4408064-svc", component="backend", name="azuredeploymentstackdesires", region="canadacentral", severity="warning" Description: Backend controller workqueue azuredeploymentstackdesires has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 1.26%3 of 238 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w9r5khxz8ndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xb6e947e0480>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w9r5khxz8ndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j5930880-mgmt-1/ocm-arohcpci01-2tv796dpjff31l6feck2okmv8t772jtv-cluster router=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003: expected matching summary after test start and within 8m; summaryUID=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003 pass=cd51b6a3-caeb-4154-b1f9-eba63e5bd4ae at=2026-11-08T18:59:11Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w9r5khxz8ndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j5930880-mgmt-1/ocm-arohcpci01-2tv796dpjff31l6feck2okmv8t772jtv-cluster router=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003: expected matching summary after test start and within 8m; summaryUID=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003 pass=cd51b6a3-caeb-4154-b1f9-eba63e5bd4ae at=2026-11-08T18:59:11Z",
err: <*errors.errorString | 0xb6e95934110>{
s: "dev/westus3/ci01-j5930880-mgmt-1/ocm-arohcpci01-2tv796dpjff31l6feck2okmv8t772jtv-cluster router=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003: expected matching summary after test start and within 8m; summaryUID=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003 pass=cd51b6a3-caeb-4154-b1f9-eba63e5bd4ae at=2026-11-08T18:59:11Z",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w9r5khxz8ndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xb6e947e0480>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w9r5khxz8ndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j5930880-mgmt-1/ocm-arohcpci01-2tv796dpjff31l6feck2okmv8t772jtv-cluster router=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003: expected matching summary after test start and within 8m; summaryUID=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003 pass=cd51b6a3-caeb-4154-b1f9-eba63e5bd4ae at=2026-11-08T18:59:11Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w9r5khxz8ndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j5930880-mgmt-1/ocm-arohcpci01-2tv796dpjff31l6feck2okmv8t772jtv-cluster router=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003: expected matching summary after test start and within 8m; summaryUID=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003 pass=cd51b6a3-caeb-4154-b1f9-eba63e5bd4ae at=2026-11-08T18:59:11Z",
err: <*errors.errorString | 0xb6e95934110>{
s: "dev/westus3/ci01-j5930880-mgmt-1/ocm-arohcpci01-2tv796dpjff31l6feck2okmv8t772jtv-cluster router=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003: expected matching summary after test start and within 8m; summaryUID=17ae2173-30a6-4fe5-8978-b0e596577b9a node=aks-userswft3-42361884-vmss000003 pass=cd51b6a3-caeb-4154-b1f9-eba63e5bd4ae at=2026-11-08T18:59:11Z",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qkmcdqgl2brv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xa7b6deb83e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qkmcdqgl2brv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/canadacentral/ci01-j2070528-mgmt-2/ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qkmcdqgl2brv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/canadacentral/ci01-j2070528-mgmt-2/ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0xa7b6deb83c0>{
observation: <*errors.errorString | 0xa7b6cf5bba0>{
s: "dev/canadacentral/ci01-j2070528-mgmt-2/ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0xa7b6deb83a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2070528-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster\"; let start = datetime(2026-11-06T10:52:25.2955806Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0xa7b6dbd89b0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0xa7b6deb8380>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2070528-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster\"; let start = datetime(2026-11-06T10:52:25.2955806Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0xa7b6e156a80>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qkmcdqgl2brv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0xa7b6deb83e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qkmcdqgl2brv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/canadacentral/ci01-j2070528-mgmt-2/ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qkmcdqgl2brv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/canadacentral/ci01-j2070528-mgmt-2/ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
err: <*verifiers.pollObservationError | 0xa7b6deb83c0>{
observation: <*errors.errorString | 0xa7b6cf5bba0>{
s: "dev/canadacentral/ci01-j2070528-mgmt-2/ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
},
interruption: <*fmt.wrapError | 0xa7b6deb83a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2070528-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster\"; let start = datetime(2026-11-06T10:52:25.2955806Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0xa7b6dbd89b0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0xa7b6deb8380>{
msg: "With query: let env = \"dev\"; let rgn = \"canadacentral\"; let cl = \"ci01-j2070528-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttm7am0bqbqqtd1eu55gfufun1rh5ne-cluster\"; let start = datetime(2026-11-06T10:52:25.2955806Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0xa7b6e156a80>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ktzf7cfb8tpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1c793938a4c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ktzf7cfb8tpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j9600896-mgmt-2/ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster router=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001: expected matching summary after test start and within 8m; summaryUID=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001 pass=d6207ada-fb0a-45ac-8eb8-00cd79beac99 at=2026-11-06T09:27:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ktzf7cfb8tpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j9600896-mgmt-2/ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster router=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001: expected matching summary after test start and within 8m; summaryUID=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001 pass=d6207ada-fb0a-45ac-8eb8-00cd79beac99 at=2026-11-06T09:27:59Z",
err: <*verifiers.pollObservationError | 0x1c793938a480>{
observation: <*errors.errorString | 0x1c79394e38d0>{
s: "dev/westus3/ci01-j9600896-mgmt-2/ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster router=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001: expected matching summary after test start and within 8m; summaryUID=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001 pass=d6207ada-fb0a-45ac-8eb8-00cd79beac99 at=2026-11-06T09:27:59Z",
},
interruption: <*fmt.wrapError | 0x1c793938a460>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9600896-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster\"; let start = datetime(2026-11-06T09:10:17.6777034Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1c79382341e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1c793938a440>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9600896-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster\"; let start = datetime(2026-11-06T09:10:17.6777034Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1c793a5ad290>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ktzf7cfb8tpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1c793938a4c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ktzf7cfb8tpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j9600896-mgmt-2/ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster router=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001: expected matching summary after test start and within 8m; summaryUID=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001 pass=d6207ada-fb0a-45ac-8eb8-00cd79beac99 at=2026-11-06T09:27:59Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-ktzf7cfb8tpj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.01s: dev/westus3/ci01-j9600896-mgmt-2/ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster router=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001: expected matching summary after test start and within 8m; summaryUID=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001 pass=d6207ada-fb0a-45ac-8eb8-00cd79beac99 at=2026-11-06T09:27:59Z",
err: <*verifiers.pollObservationError | 0x1c793938a480>{
observation: <*errors.errorString | 0x1c79394e38d0>{
s: "dev/westus3/ci01-j9600896-mgmt-2/ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster router=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001: expected matching summary after test start and within 8m; summaryUID=0dcbed7e-d497-41e8-a495-cfc1f0cc0d71 node=aks-userswft3-22872461-vmss000001 pass=d6207ada-fb0a-45ac-8eb8-00cd79beac99 at=2026-11-06T09:27:59Z",
},
interruption: <*fmt.wrapError | 0x1c793938a460>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9600896-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster\"; let start = datetime(2026-11-06T09:10:17.6777034Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1c79382341e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1c793938a440>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9600896-mgmt-2\"; let ns = \"ocm-arohcpci01-2ttknfvkoiplih49juot6ftjifd2gkkj-cluster\"; let start = datetime(2026-11-06T09:10:17.6777034Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1c793a5ad290>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 80.00% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster then delete it by deleting the customer resource group | |
| 84.44% | 45 | integration/parallel | Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully | |
| 84.44% | 45 | integration/parallel | Customer should be able to update node pool labels and taints |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InvalidResourceType; detail message The resource type 'HCPOpenShiftClusters' could not be found in the namespace 'Microsoft.RedHatOpenShift' for api version '2027-02-0...full failure pattern: ERROR CODE: InvalidResourceType; detail message The resource type 'HCPOpenShiftClusters' could not be found in the namespace 'Microsoft.RedHatOpenShift' for api version '2027-02-01'. The supported api-versions are '2026-10-01,2026-09-01-preview,2026-06-30-preview,2025-12-23-preview,2024-06-10-preview'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 2 | 9.09%2 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | |||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20270201.go:51]: failed to probe v20270201 API availability
Unexpected error:
<*exported.ResponseError | 0x13496d5a43f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: InvalidResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidResourceType",
"message": "The resource type 'HCPOpenShiftClusters' could not be found in the namespace 'Microsoft.RedHatOpenShift' for api version '2027-02-01'. The supported api-versions are '2026-10-01,2026-09-01-preview,2026-06-30-preview,2025-12-23-preview,2024-06-10-preview'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: d2341063-4b8f-4a0b-9e2b-18a4f5742efd
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20270201.go:51]: failed to probe v20270201 API availability
Unexpected error:
<*exported.ResponseError | 0x13496d5a43f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: InvalidResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidResourceType",
"message": "The resource type 'HCPOpenShiftClusters' could not be found in the namespace 'Microsoft.RedHatOpenShift' for api version '2027-02-01'. The supported api-versions are '2026-10-01,2026-09-01-preview,2026-06-30-preview,2025-12-23-preview,2024-06-10-preview'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: d2341063-4b8f-4a0b-9e2b-18a4f5742efd
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20270201.go:51]: failed to probe v20270201 API availability
Unexpected error:
<*exported.ResponseError | 0x1c96dfa50210>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: InvalidResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidResourceType",
"message": "The resource type 'HCPOpenShiftClusters' could not be found in the namespace 'Microsoft.RedHatOpenShift' for api version '2027-02-01'. The supported api-versions are '2026-10-01,2026-09-01-preview,2026-06-30-preview,2025-12-23-preview,2024-06-10-preview'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: a6b3915b-22f1-4e3a-aa32-2cc3ca8dd99b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20270201.go:51]: failed to probe v20270201 API availability
Unexpected error:
<*exported.ResponseError | 0x1c96dfa50210>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: InvalidResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidResourceType",
"message": "The resource type 'HCPOpenShiftClusters' could not be found in the namespace 'Microsoft.RedHatOpenShift' for api version '2027-02-01'. The supported api-versions are '2026-10-01,2026-09-01-preview,2026-06-30-preview,2025-12-23-preview,2024-06-10-preview'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters
correlation ID: a6b3915b-22f1-4e3a-aa32-2cc3ca8dd99b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | DEV | |||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-11-05T04:15:54Z"}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceededfull failure pattern: DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | |||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x3311a2f0c120>:
resource group "rg-candidate-4-23-kk2s78nmqwx7": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7", caused by: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x3311a20263e0>{
msg: "resource group \"rg-candidate-4-23-kk2s78nmqwx7\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapError | 0x3311a20263c0>{
msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapErrors | 0x3311a23a85a0>{
msg: "failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*fmt.wrapErrors | 0x3311a23a8570>{
msg: "failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3311a2366650>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7",
},
<context.deadlineExceededError>{},
],
},
<*fmt.wrapErrors | 0x3311a23a8570>0x3311a23a8570 (cyclic reference),
],
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x3311a2f0c120>:
resource group "rg-candidate-4-23-kk2s78nmqwx7": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7", caused by: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-23-2r6jdv" in resourcegroup="rg-candidate-4-23-kk2s78nmqwx7" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x3311a20263e0>{
msg: "resource group \"rg-candidate-4-23-kk2s78nmqwx7\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapError | 0x3311a20263c0>{
msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
err: <*fmt.wrapErrors | 0x3311a23a85a0>{
msg: "failed deleting hcp clusters in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\", caused by: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*fmt.wrapErrors | 0x3311a23a8570>{
msg: "failed waiting for hcpCluster=\"cluster-candidate-4-23-2r6jdv\" in resourcegroup=\"rg-candidate-4-23-kk2s78nmqwx7\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3311a2366650>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-23-2r6jdv in resource group rg-candidate-4-23-kk2s78nmqwx7",
},
<context.deadlineExceededError>{},
],
},
<*fmt.wrapErrors | 0x3311a23a8570>0x3311a23a8570 (cyclic reference),
],
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining res...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | |||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x142a3dd04180>:
resource group "rg-candidate-4-23-cxmw6dwj8lvz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-l5472k" in resourcegroup="rg-candidate-4-23-cxmw6dwj8lvz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"name": "9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"status": "Failed",
"startTime": "2026-11-09T17:35:42.053088057Z",
"endTime": "2026-11-09T17:59:50.562259882Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
correlation ID: 97bda22f-2f83-4bf1-94bb-3865756e10f7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x142a3dd04180>:
resource group "rg-candidate-4-23-cxmw6dwj8lvz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-l5472k" in resourcegroup="rg-candidate-4-23-cxmw6dwj8lvz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"name": "9448b2d8-49dd-4e89-9bb5-597589d2b5c3",
"status": "Failed",
"startTime": "2026-11-09T17:35:42.053088057Z",
"endTime": "2026-11-09T17:59:50.562259882Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9448b2d8-49dd-4e89-9bb5-597589d2b5c3
correlation ID: 97bda22f-2f83-4bf1-94bb-3865756e10f7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 76.92% | 13 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 76.92% | 13 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 76.92% | 13 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 76.92% | 13 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 76.92% | 13 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 9.09%2 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 2 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | PROD | |||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x36f572541e00>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: d99ae92f-346f-4c80-93a7-e1a36af3cb11
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x36f572541e00>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: d99ae92f-346f-4c80-93a7-e1a36af3cb11
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x3a0d38f1bd10>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: b9451470-1a23-4058-8ee4-b97dbe665919
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x3a0d38f1bd10>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: b9451470-1a23-4058-8ee4-b97dbe665919
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x28d3bbcb0540>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: 0fca0e5b-21cb-48a3-99c9-d833509498cb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x28d3bbcb0540>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperatorIdentityRoleSets
correlation ID: 0fca0e5b-21cb-48a3-99c9-d833509498cb
client request ID: unavailable
occurredContributing tests (4)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator, ingress-operator]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | |||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x270039317a40>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-q2hwcpbx7mqn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220",
"name": "2f19f972-635b-47fe-bd8e-1577e0895220",
"status": "Failed",
"startTime": "2026-11-08T23:05:43.229286441Z",
"endTime": "2026-11-08T23:34:48.983552333Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
correlation ID: 2b542b89-9e3f-45d8-a8b2-acddb890ebb9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x270039317a40>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-q2hwcpbx7mqn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220",
"name": "2f19f972-635b-47fe-bd8e-1577e0895220",
"status": "Failed",
"startTime": "2026-11-08T23:05:43.229286441Z",
"endTime": "2026-11-08T23:34:48.983552333Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [cluster-network-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f19f972-635b-47fe-bd8e-1577e0895220
correlation ID: 2b542b89-9e3f-45d8-a8b2-acddb890ebb9
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | |||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x5b2334e0700>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wm9dc9s5xztk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153",
"name": "12b091af-f05e-4977-a8fb-00833f004153",
"status": "Failed",
"startTime": "2026-11-08T23:07:24.292926055Z",
"endTime": "2026-11-08T23:36:29.334124474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 11m56s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
correlation ID: ca2f27d7-78cd-47db-9589-31ad832f7406
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x5b2334e0700>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wm9dc9s5xztk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153",
"name": "12b091af-f05e-4977-a8fb-00833f004153",
"status": "Failed",
"startTime": "2026-11-08T23:07:24.292926055Z",
"endTime": "2026-11-08T23:36:29.334124474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 11m56s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12b091af-f05e-4977-a8fb-00833f004153
correlation ID: ca2f27d7-78cd-47db-9589-31ad832f7406
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 1 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | PROD | |||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:105]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x8ede2e071d0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x8ede274bbe0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x8ede274bbc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x8ede315d2c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:105]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x8ede2e071d0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x8ede274bbe0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x8ede274bbc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x8ede315d2c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "c...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xdkrt6-zjdgs-44h72": nodes "cluster-np-xdkrt6-zjdgs-44h72" is Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 1 · Nov 10: 0 · Nov 11: 0 | none | |||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xdkrt6"
Unexpected error:
<*errors.joinError | 0x22f6aa118348>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xdkrt6-zjdgs-44h72": nodes "cluster-np-xdkrt6-zjdgs-44h72" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x22f6a8c83260>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83240>{
msg: "failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83220>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x22f6a8ca6fa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xdkrt6-zjdgs-44h72",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:297]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xdkrt6"
Unexpected error:
<*errors.joinError | 0x22f6aa118348>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xdkrt6-zjdgs-44h72": nodes "cluster-np-xdkrt6-zjdgs-44h72" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x22f6a8c83260>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83240>{
msg: "failed to label node cluster-np-xdkrt6-zjdgs-44h72 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x22f6a8c83220>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xdkrt6-zjdgs-44h72\": nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x22f6a8ca6fa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xdkrt6-zjdgs-44h72\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xdkrt6-zjdgs-44h72",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 91.67% | 12 | prod/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 91.67% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 91.67% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 91.67% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 91.67% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: NotFound; detail message The requested path could not be found.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | e2e | 46 | 50.55%46 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsNov 4: 0 · Nov 5: 4 · Nov 6: 68 · Nov 7: 10 · Nov 8: 8 · Nov 9: 2 · Nov 10: 0 · Nov 11: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e16e18c95f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: e5e9735e-d3e5-4012-86ee-338d0c182df4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e16e18c95f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: e5e9735e-d3e5-4012-86ee-338d0c182df4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2aad74a796b0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: 38cd85f4-6402-47f3-975b-00cf55650f28
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2aad74a796b0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperatorIdentityRoleSets
correlation ID: 38cd85f4-6402-47f3-975b-00cf55650f28
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x364c7b3b3050>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 7f875e35-2e2b-44ef-9cc5-b907e9231d1e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x364c7b3b3050>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "The requested path could not be found."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperatorIdentityRoleSets
correlation ID: 7f875e35-2e2b-44ef-9cc5-b907e9231d1e
client request ID: unavailable
occurredContributing tests (6)
Affected runs (46) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | e2e | 5 | 5.49%5 of 91 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsNov 4: 2 · Nov 5: 1 · Nov 6: 3 · Nov 7: 1 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x32945d3b9620>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mpc4jqncvt9z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3",
"name": "4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3",
"status": "Failed",
"startTime": "2026-11-07T02:12:27.722960469Z",
"endTime": "2026-11-07T02:31:27.176787895Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3
correlation ID: d2ff85b0-6cfb-445f-b483-c95ae69c981e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x32945d3b9620>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mpc4jqncvt9z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3",
"name": "4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3",
"status": "Failed",
"startTime": "2026-11-07T02:12:27.722960469Z",
"endTime": "2026-11-07T02:31:27.176787895Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4e1c85e0-b8c8-4c0a-9d1e-8320e6c568a3
correlation ID: d2ff85b0-6cfb-445f-b483-c95ae69c981e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x32e880a2f1e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-md8ff7h8pp9h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d2909866-e47a-418c-9911-d4216b9a2b69
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d2909866-e47a-418c-9911-d4216b9a2b69",
"name": "d2909866-e47a-418c-9911-d4216b9a2b69",
"status": "Failed",
"startTime": "2026-11-06T23:00:55.172608282Z",
"endTime": "2026-11-06T23:20:04.210498067Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d2909866-e47a-418c-9911-d4216b9a2b69
correlation ID: 7c92fe2e-686f-47be-8a63-2250bf2366cd
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x32e880a2f1e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-md8ff7h8pp9h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d2909866-e47a-418c-9911-d4216b9a2b69
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d2909866-e47a-418c-9911-d4216b9a2b69",
"name": "d2909866-e47a-418c-9911-d4216b9a2b69",
"status": "Failed",
"startTime": "2026-11-06T23:00:55.172608282Z",
"endTime": "2026-11-06T23:20:04.210498067Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d2909866-e47a-418c-9911-d4216b9a2b69
correlation ID: 7c92fe2e-686f-47be-8a63-2250bf2366cd
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x33e8b8c86c60>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-56s2q6lfx9lj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/150e763f-b801-4b9d-b09a-01212adc492d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/150e763f-b801-4b9d-b09a-01212adc492d",
"name": "150e763f-b801-4b9d-b09a-01212adc492d",
"status": "Failed",
"startTime": "2026-11-06T20:18:37.484988551Z",
"endTime": "2026-11-06T20:37:40.939176326Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/150e763f-b801-4b9d-b09a-01212adc492d
correlation ID: 2ece0651-fe2f-487a-826f-d2951f3493a0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x33e8b8c86c60>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-56s2q6lfx9lj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/150e763f-b801-4b9d-b09a-01212adc492d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/150e763f-b801-4b9d-b09a-01212adc492d",
"name": "150e763f-b801-4b9d-b09a-01212adc492d",
"status": "Failed",
"startTime": "2026-11-06T20:18:37.484988551Z",
"endTime": "2026-11-06T20:37:40.939176326Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/150e763f-b801-4b9d-b09a-01212adc492d
correlation ID: 2ece0651-fe2f-487a-826f-d2951f3493a0
client request ID: unavailable
occurredContributing tests (1)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or q...full failure pattern: selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 4 | 4.40%4 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 8 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:92]: failed to find an ARM64-capable VM size; check VM SKU restrictions/quota for the test subscription in eastus2
Unexpected error:
<*fmt.wrapError | 0x215578d78ea0>:
selector "arm64-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"arm64-worker\" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x172c4c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:92]: failed to find an ARM64-capable VM size; check VM SKU restrictions/quota for the test subscription in eastus2
Unexpected error:
<*fmt.wrapError | 0x215578d78ea0>:
selector "arm64-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"arm64-worker\" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x172c4c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
<*fmt.wrapError | 0x1378e7a92ba0>:
selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"ephemeral-osdisk-worker\" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x172c4c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
<*fmt.wrapError | 0x1378e7a92ba0>:
selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"ephemeral-osdisk-worker\" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x172c4c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:92]: failed to find an ARM64-capable VM size; check VM SKU restrictions/quota for the test subscription in eastus2
Unexpected error:
<*fmt.wrapError | 0x1c7eea5e0c20>:
selector "arm64-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"arm64-worker\" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x172c4c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:92]: failed to find an ARM64-capable VM size; check VM SKU restrictions/quota for the test subscription in eastus2
Unexpected error:
<*fmt.wrapError | 0x1c7eea5e0c20>:
selector "arm64-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
{
msg: "selector \"arm64-worker\" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
err: <*errors.errorString | 0x172c4c20>{
s: "no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect",
},
}
occurredContributing tests (2)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.20%2 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-4-22-brdgjq" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x16cd1072ca98>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x16cd10c558e0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x16cd10c558c0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x16cd1007fc20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x16cd0fed5720>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x16cd0fed5700>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x16cd1033cdc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-4-22-brdgjq" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x16cd1072ca98>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x16cd10c558e0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x16cd10c558c0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x16cd1007fc20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x16cd0fed5720>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x16cd0fed5700>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x16cd1033cdc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-4l8tt7" is viable
Unexpected error:
<*errors.joinError | 0x538c7490030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x538c6e1c1a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x538c6e1c180>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x538c70ecdc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x538c669edc0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x538c669ebc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x538c6e7a6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-4l8tt7" is viable
Unexpected error:
<*errors.joinError | 0x538c7490030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x538c6e1c1a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x538c6e1c180>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x538c70ecdc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x538c669edc0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x538c669ebc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x538c6e7a6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 2 | 2.20%2 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 0 · Nov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
<*errors.joinError | 0x21c2e72b6dc8>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21c2e7d1a4e0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21c2e7d1a4c0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21c2e7312280>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21c2e8078100>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21c2e80780e0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21c2e7732f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
<*errors.joinError | 0x21c2e72b6dc8>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21c2e7d1a4e0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21c2e7d1a4c0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21c2e7312280>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21c2e8078100>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21c2e80780e0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21c2e7732f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:199]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0xdd566cf84c8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0xdd567282d20>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0xdd567282d00>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0xdd5670ae5a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:199]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0xdd566cf84c8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0xdd567282d20>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0xdd567282d00>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0xdd5670ae5a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
storage account log verification failedfull failure pattern: storage account log verification failed Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.20%2 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 4: 1 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxtqqdwx and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-whkjvq5vlpvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
<*errors.errorString | 0x1d329eecd060>:
storage account log verification failed
{
s: "storage account log verification failed",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxtqqdwx and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-whkjvq5vlpvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
<*errors.errorString | 0x1d329eecd060>:
storage account log verification failed
{
s: "storage account log verification failed",
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxc8kvjg and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-wbgq6w7vjsl4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
<*errors.errorString | 0x9dfb379f3d0>:
storage account log verification failed
{
s: "storage account log verification failed",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxc8kvjg and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-wbgq6w7vjsl4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
<*errors.errorString | 0x9dfb379f3d0>:
storage account log verification failed
{
s: "storage account log verification failed",
}
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft....full failure pattern: ERROR CODE: DeadlineExceeded; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/<resource>' over scope '/subscriptions/<subscription>/resour... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.20%2 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x5b6516e6a60>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-nkkgr6vcj2gb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ac91c7c-e889-4831-8d19-dd5b8df7ff99
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ac91c7c-e889-4831-8d19-dd5b8df7ff99",
"name": "7ac91c7c-e889-4831-8d19-dd5b8df7ff99",
"status": "Failed",
"startTime": "2026-11-06T21:04:07.156298388Z",
"endTime": "2026-11-06T21:23:14.159748063Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-arm64-vm-np-1-hf7bs-4brq9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-arm64-vm-np-1-hf7bs-4brq9: Failed: virtualmachine failed to create or update. err: failed to reapply VM arm64-vm-cluster-nkkgr6vcj2gb--managed/cluster-arm64-vm-np-1-hf7bs-4brq9: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-nkkgr6vcj2gb--managed/providers/Microsoft.Compute/virtualMachines/cluster-arm64-vm-np-1-hf7bs-4brq9/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '208d66aa-949f-41ca-ba17-87964abb5674' with object id 'f25d7383-ad30-4e6d-bb16-fa2ba4b5ec83' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resour... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ac91c7c-e889-4831-8d19-dd5b8df7ff99
correlation ID: 7cee2cab-545b-4f8a-bfac-bdea62f05707
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x5b6516e6a60>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-nkkgr6vcj2gb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ac91c7c-e889-4831-8d19-dd5b8df7ff99
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ac91c7c-e889-4831-8d19-dd5b8df7ff99",
"name": "7ac91c7c-e889-4831-8d19-dd5b8df7ff99",
"status": "Failed",
"startTime": "2026-11-06T21:04:07.156298388Z",
"endTime": "2026-11-06T21:23:14.159748063Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-arm64-vm-np-1-hf7bs-4brq9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-arm64-vm-np-1-hf7bs-4brq9: Failed: virtualmachine failed to create or update. err: failed to reapply VM arm64-vm-cluster-nkkgr6vcj2gb--managed/cluster-arm64-vm-np-1-hf7bs-4brq9: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-nkkgr6vcj2gb--managed/providers/Microsoft.Compute/virtualMachines/cluster-arm64-vm-np-1-hf7bs-4brq9/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '208d66aa-949f-41ca-ba17-87964abb5674' with object id 'f25d7383-ad30-4e6d-bb16-fa2ba4b5ec83' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resour... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7ac91c7c-e889-4831-8d19-dd5b8df7ff99
correlation ID: 7cee2cab-545b-4f8a-bfac-bdea62f05707
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x3a25dd21ece0>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jgk4k9gqw6gh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/409bb4ee-f4ae-416d-8d78-78ccdafd6731
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/409bb4ee-f4ae-416d-8d78-78ccdafd6731",
"name": "409bb4ee-f4ae-416d-8d78-78ccdafd6731",
"status": "Failed",
"startTime": "2026-11-06T09:07:03.834317482Z",
"endTime": "2026-11-06T09:26:07.456760707Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-arm64-vm-np-1-khql6-k5h2v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-arm64-vm-np-1-khql6-k5h2v: Failed: virtualmachine failed to create or update. err: failed to reapply VM arm64-vm-cluster-jgk4k9gqw6gh--managed/cluster-arm64-vm-np-1-khql6-k5h2v: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jgk4k9gqw6gh--managed/providers/Microsoft.Compute/virtualMachines/cluster-arm64-vm-np-1-khql6-k5h2v/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'a11420bd-9bf9-4662-b795-cbd1719d619a' with object id 'f5464816-f9ea-45e8-91e9-4f58e7f9bae7' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resour... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/409bb4ee-f4ae-416d-8d78-78ccdafd6731
correlation ID: 72c9280f-8623-45a1-aac1-95b6a4ff5eff
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x3a25dd21ece0>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jgk4k9gqw6gh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/409bb4ee-f4ae-416d-8d78-78ccdafd6731
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/409bb4ee-f4ae-416d-8d78-78ccdafd6731",
"name": "409bb4ee-f4ae-416d-8d78-78ccdafd6731",
"status": "Failed",
"startTime": "2026-11-06T09:07:03.834317482Z",
"endTime": "2026-11-06T09:26:07.456760707Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-arm64-vm-np-1-khql6-k5h2v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-arm64-vm-np-1-khql6-k5h2v: Failed: virtualmachine failed to create or update. err: failed to reapply VM arm64-vm-cluster-jgk4k9gqw6gh--managed/cluster-arm64-vm-np-1-khql6-k5h2v: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jgk4k9gqw6gh--managed/providers/Microsoft.Compute/virtualMachines/cluster-arm64-vm-np-1-khql6-k5h2v/reapply\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'a11420bd-9bf9-4662-b795-cbd1719d619a' with object id 'f5464816-f9ea-45e8-91e9-4f58e7f9bae7' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/reapply/action' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resour... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/409bb4ee-f4ae-416d-8d78-78ccdafd6731
correlation ID: 72c9280f-8623-45a1-aac1-95b6a4ff5eff
client request ID: unavailable
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...full failure pattern: ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.20%2 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 1 · Nov 11: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
<*fmt.wrapError | 0x29acc44990c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-disabled-image-registry-hcp-cluster-gwmhk7" in resourcegroup="disabled-image-registry-6jmbtjt72wm7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 49965461-a7c3-410f-b49e-8883df244823",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-tez7gwgoxyw3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
correlation ID: 9da5bde4-a864-47a7-bbec-d7354cb68d5b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
<*fmt.wrapError | 0x29acc44990c0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-disabled-image-registry-hcp-cluster-gwmhk7" in resourcegroup="disabled-image-registry-6jmbtjt72wm7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 49965461-a7c3-410f-b49e-8883df244823",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-tez7gwgoxyw3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/disabled-image-registry-6jmbtjt72wm7/providers/Microsoft.Resources/deployments/customer-infra-disabled-image-registry-hcp-cluster-gwmhk7/operationStatuses/08584072821722419224
correlation ID: 9da5bde4-a864-47a7-bbec-d7354cb68d5b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:143]: failed to create customer resources for cluster "cluster-candidate-4-20-xw84kr"
Unexpected error:
<*fmt.wrapError | 0x2722651c160>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz" in resourcegroup="rg-candidate-4-20-zwlln5grl4vh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 5c19921f-4101-4b07-aab3-b71cf2831d22",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-6xl6wx5rad6lg' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
correlation ID: 3b996925-9aa8-423b-8b22-869e2b8ad84e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:143]: failed to create customer resources for cluster "cluster-candidate-4-20-xw84kr"
Unexpected error:
<*fmt.wrapError | 0x2722651c160>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz" in resourcegroup="rg-candidate-4-20-zwlln5grl4vh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 5c19921f-4101-4b07-aab3-b71cf2831d22",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-6xl6wx5rad6lg' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-zwlln5grl4vh/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-xw84kr-6pbwwz/operationStatuses/08584075018488209160
correlation ID: 3b996925-9aa8-423b-8b22-869e2b8ad84e
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-<id> namespace are ready: failed to list pods in cilium-connectivity-ch...full failure pattern: VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-<id> namespace are ready: failed to list pods in cilium-connectivity-check-<id> namespace: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.20%2 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 2 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x27d4135210b0>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-nwwd7l namespace are ready: failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x27d4136fab00>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-nwwd7l namespace are ready: failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x27d4136fa940>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-nwwd7l namespace are ready: failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x27d41221f900>{
msg: "failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x27d4135210b0>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-nwwd7l namespace are ready: failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x27d4136fab00>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-nwwd7l namespace are ready: failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x27d4136fa940>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-nwwd7l namespace are ready: failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x27d41221f900>{
msg: "failed to list pods in cilium-connectivity-check-nwwd7l namespace: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x21961327a690>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-gm7fc7 namespace are ready: failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x21961526f4a0>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-gm7fc7 namespace are ready: failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x2196135ab760>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-gm7fc7 namespace are ready: failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x21961526f180>{
msg: "failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x21961327a690>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-gm7fc7 namespace are ready: failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x21961526f4a0>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-gm7fc7 namespace are ready: failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x2196135ab760>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-gm7fc7 namespace are ready: failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded",
err: <*fmt.wrapError | 0x21961526f180>{
msg: "failed to list pods in cilium-connectivity-check-gm7fc7 namespace: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 69 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x357a2fe5f040>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-csp4tk4h8qs9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: 49abcf5f-88e9-49d5-b8c2-5f1d082bc74c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x357a2fe5f040>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-csp4tk4h8qs9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-csp4tk4h8qs9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: 49abcf5f-88e9-49d5-b8c2-5f1d082bc74c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-mcx8z4" with version 4.22.20 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3e43e6d40140>:
failed to create HCP cluster cluster-zstream-4-22-mcx8z4: failed starting cluster creation "cluster-zstream-4-22-mcx8z4" in resourcegroup="rg-zstream-upgrade-4-22-s6jpsvnr8lwq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
correlation ID: a7c5e206-75d9-4088-8da5-8a945905e74d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-mcx8z4" with version 4.22.20 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3e43e6d40140>:
failed to create HCP cluster cluster-zstream-4-22-mcx8z4: failed starting cluster creation "cluster-zstream-4-22-mcx8z4" in resourcegroup="rg-zstream-upgrade-4-22-s6jpsvnr8lwq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-s6jpsvnr8lwq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-mcx8z4
correlation ID: a7c5e206-75d9-4088-8da5-8a945905e74d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xeef026fef40>:
failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-hsxdls4m5cf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
correlation ID: 0d18fc06-8f55-4eae-9e49-cbd4e9f7546f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xeef026fef40>:
failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-hsxdls4m5cf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hsxdls4m5cf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
correlation ID: 0d18fc06-8f55-4eae-9e49-cbd4e9f7546f
client request ID: unavailable
occurredContributing tests (69)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClu...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 69 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x14691311a640>:
failed to create HCP cluster np-downgrade-4gmdjb: failed starting cluster creation "np-downgrade-4gmdjb" in resourcegroup="rg-np-downgrade-4gmdjb-cl9h9rwwkkx6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-4gmdjb-cl9h9rwwkkx6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-4gmdjb
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-4gmdjb-cl9h9rwwkkx6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-4gmdjb
correlation ID: 6cfda46d-97d6-449e-aa66-761b97df8f6a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x14691311a640>:
failed to create HCP cluster np-downgrade-4gmdjb: failed starting cluster creation "np-downgrade-4gmdjb" in resourcegroup="rg-np-downgrade-4gmdjb-cl9h9rwwkkx6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-4gmdjb-cl9h9rwwkkx6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-4gmdjb
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-4gmdjb-cl9h9rwwkkx6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-4gmdjb
correlation ID: 6cfda46d-97d6-449e-aa66-761b97df8f6a
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-rqsd7mwp4t6p/cluster-candidate-4-21-g6dw85 should provision
Unexpected error:
<*fmt.wrapError | 0xc2cf21bc2c0>:
failed to create HCP cluster cluster-candidate-4-21-g6dw85: failed starting cluster creation "cluster-candidate-4-21-g6dw85" in resourcegroup="rg-candidate-4-21-rqsd7mwp4t6p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rqsd7mwp4t6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-g6dw85
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rqsd7mwp4t6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-g6dw85
correlation ID: 221991e3-f696-47a7-8bd1-833fd02c103a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-rqsd7mwp4t6p/cluster-candidate-4-21-g6dw85 should provision
Unexpected error:
<*fmt.wrapError | 0xc2cf21bc2c0>:
failed to create HCP cluster cluster-candidate-4-21-g6dw85: failed starting cluster creation "cluster-candidate-4-21-g6dw85" in resourcegroup="rg-candidate-4-21-rqsd7mwp4t6p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rqsd7mwp4t6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-g6dw85
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-rqsd7mwp4t6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-g6dw85
correlation ID: 221991e3-f696-47a7-8bd1-833fd02c103a
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0xc355bcf89a0>:
failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-bhtt9gv7ttlh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bhtt9gv7ttlh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bhtt9gv7ttlh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
correlation ID: 5f23ac5b-8295-4908-8db4-ea752bee9bfb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0xc355bcf89a0>:
failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-bhtt9gv7ttlh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bhtt9gv7ttlh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bhtt9gv7ttlh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
correlation ID: 5f23ac5b-8295-4908-8db4-ea752bee9bfb
client request ID: unavailable
occurredContributing tests (69)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
fetching manifest <url>full failure pattern: fetching manifest <url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 2 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x320cc6eca0e0>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x320cc6882108>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x320cc655b860>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x320cc6eca060>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x320cc60b1250>{s: "HTTP 504"},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x320cc6eca0e0>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x320cc6882108>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x320cc655b860>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x320cc6eca060>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x320cc60b1250>{s: "HTTP 504"},
},
],
},
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x18ddcdc0fa40>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x18ddcdc24eb8>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x18ddcd102320>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x18ddcdc0fa00>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x18ddcf55f050>{s: "HTTP 504"},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x18ddcdc0fa40>:
fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition
last download error: HTTP 504
{
msg: "fetching manifest https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml after 5 attempts: timed out waiting for the condition\nlast download error: HTTP 504",
err: <*errors.joinError | 0x18ddcdc24eb8>{
errs: [
<wait.errInterrupted>{
cause: <*errors.errorString | 0x18ddcd102320>{
s: "timed out waiting for the condition",
},
},
<*fmt.wrapError | 0x18ddcdc0fa00>{
msg: "last download error: HTTP 504",
err: <*errors.errorString | 0x18ddcf55f050>{s: "HTTP 504"},
},
],
},
}Contributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 4: 1 · Nov 5: 0 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
<*fmt.wrapError | 0x1082dba9ec00>:
failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-j88hb8f2qk7s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f12f8f4-d169-4766-83ad-cd6879f4785d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f12f8f4-d169-4766-83ad-cd6879f4785d",
"name": "4f12f8f4-d169-4766-83ad-cd6879f4785d",
"status": "Failed",
"startTime": "2026-11-06T12:47:38.256626Z",
"endTime": "2026-11-06T13:06:47.051014443Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-des-np-rqdb4-9r7vm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-des-np-rqdb4-k8dr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-des-np-rqdb4-9r7vm: Creating: virtualmachine creating or updating\nMachine cluster-des-np-rqdb4-k8dr6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f12f8f4-d169-4766-83ad-cd6879f4785d
correlation ID: b45d1d87-a775-49c8-afdd-bd13377b6db7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
<*fmt.wrapError | 0x1082dba9ec00>:
failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-j88hb8f2qk7s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f12f8f4-d169-4766-83ad-cd6879f4785d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f12f8f4-d169-4766-83ad-cd6879f4785d",
"name": "4f12f8f4-d169-4766-83ad-cd6879f4785d",
"status": "Failed",
"startTime": "2026-11-06T12:47:38.256626Z",
"endTime": "2026-11-06T13:06:47.051014443Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-des-np-rqdb4-9r7vm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-des-np-rqdb4-k8dr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-des-np-rqdb4-9r7vm: Creating: virtualmachine creating or updating\nMachine cluster-des-np-rqdb4-k8dr6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f12f8f4-d169-4766-83ad-cd6879f4785d
correlation ID: b45d1d87-a775-49c8-afdd-bd13377b6db7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 1 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0x37b08a783e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-m8jlgbs48ts9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"name": "fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"status": "Failed",
"startTime": "2026-11-08T02:36:57.421300337Z",
"endTime": "2026-11-08T02:56:03.454435501Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
correlation ID: b0decc5d-318d-423b-aa5d-75e7753c16f6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0x37b08a783e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-m8jlgbs48ts9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"name": "fa1455df-4b0e-49b6-b056-3db1ed0032e9",
"status": "Failed",
"startTime": "2026-11-08T02:36:57.421300337Z",
"endTime": "2026-11-08T02:56:03.454435501Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/fa1455df-4b0e-49b6-b056-3db1ed0032e9
correlation ID: b0decc5d-318d-423b-aa5d-75e7753c16f6
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentityfull failure pattern: ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentity Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 1 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:158]: failed to create customer resources for patch-tags cluster
Unexpected error:
<*fmt.wrapError | 0x8054d89efc0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-tlq6j2" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20261110T031257Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '8a115947-73d2-4084-928a-c2672e44e2fc', request correlation id '28d599ba-f561-4e3b-af9c-cf080eee62d4'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
correlation ID: 28a0fd91-bacb-4401-9842-5185b0320dbf
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:158]: failed to create customer resources for patch-tags cluster
Unexpected error:
<*fmt.wrapError | 0x8054d89efc0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-tlq6j2" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 28d599ba-f561-4e3b-af9c-cf080eee62d4\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-prod-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20261110T031257Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '8a115947-73d2-4084-928a-c2672e44e2fc', request correlation id '28d599ba-f561-4e3b-af9c-cf080eee62d4'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-tlq6j2/operationStatuses/08584073265740898275
correlation ID: 28a0fd91-bacb-4401-9842-5185b0320dbf
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpe...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xca03c226ff0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 43ef136d-3363-4bff-be35-8603a255e4de
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xca03c226ff0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 43ef136d-3363-4bff-be35-8603a255e4de
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/h...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0x16e09129d020>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
correlation ID: 1529b2d7-3e5f-4f86-80ce-30238c881e0d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0x16e09129d020>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
correlation ID: 1529b2d7-3e5f-4f86-80ce-30238c881e0d
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-<id> namespace are ready: test pods not yet ready: [host-to-b-multi-nod...full failure pattern: VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-<id> namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-<id> (container host-to-b-multi-node-clusterip-container waiting Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x251c31228228>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-l556fj namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]
{
errs: [
<*fmt.wrapError | 0x251c32810400>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-l556fj namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]",
err: <*fmt.wrapError | 0x251c318354c0>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-l556fj namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]",
err: <*errors.errorString | 0x251c32942060>{
s: "test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]",
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x251c31228228>:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-l556fj namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]
{
errs: [
<*fmt.wrapError | 0x251c32810400>{
msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-l556fj namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]",
err: <*fmt.wrapError | 0x251c318354c0>{
msg: "connectivity check failed, not all pods in cilium-connectivity-check-l556fj namespace are ready: test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]",
err: <*errors.errorString | 0x251c32942060>{
s: "test pods not yet ready: [host-to-b-multi-node-clusterip-7b59cb4448-7bh9x (container host-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff) host-to-b-multi-node-headless-dc99f9896-fpn2n (container host-to-b-multi-node-headless-container waiting: CrashLoopBackOff) pod-to-a-5b5d56bbc9-ft2w8 (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-fk6vp (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-clusterip-78db755458-v87m4 (container pod-to-b-multi-node-clusterip-container waiting: CrashLoopBackOff)]",
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
creating Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "<url>full failure pattern: creating Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:448]: creating Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxy error from konnectivity-server-local:8090 while dialing 10.129.0.15:10250, code 500: 500 Internal Server Error fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:448]: creating Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxy error from konnectivity-server-local:8090 while dialing 10.129.0.15:10250, code 500: 500 Internal Server Error Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
creating Certificate: Post "<url>": read tcp <ip>full failure pattern: creating Certificate: Post "<url>": read tcp <ip> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 1 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:448]: creating Certificate: Post "https://api.cluster.skl6b.eastus2euap.aroapp-hcp.io:443/apis/cert-manager.io/v1/namespaces/openshift-ingress/certificates": read tcp 10.129.49.125:51428->72.147.62.64:443: read: connection reset by peer fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:448]: creating Certificate: Post "https://api.cluster.skl6b.eastus2euap.aroapp-hcp.io:443/apis/cert-manager.io/v1/namespaces/openshift-ingress/certificates": read tcp 10.129.49.125:51428->72.147.62.64:443: read: connection reset by peer Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
error: You must be logged in to the server (the server has asked for the client to provide credentials)full failure pattern: error: You must be logged in to the server (the server has asked for the client to provide credentials) Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.10%1 of 91 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 4: 0 · Nov 5: 0 · Nov 6: 1 · Nov 7: 0 · Nov 8: 0 · Nov 9: 0 · Nov 10: 0 · Nov 11: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 321.821s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0x1eb7fdbfde30>:
E1106 14:04:48.204541 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.223010 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.243770 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.272651 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.293171 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
{
s: "E1106 14:04:48.204541 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.223010 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.243770 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.272651 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.293171 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nerror: You must be logged in to the server (the server has asked for the client to provide credentials)",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 321.821s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0x1eb7fdbfde30>:
E1106 14:04:48.204541 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.223010 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.243770 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.272651 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
E1106 14:04:48.293171 2832 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
{
s: "E1106 14:04:48.204541 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.223010 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.243770 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.272651 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nE1106 14:04:48.293171 2832 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"\nerror: You must be logged in to the server (the server has asked for the client to provide credentials)",
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||