CIHealth
<
Single Day: Mar 4▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 507eb19 · merge 507eb19
 n/a n/a passed #4290 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7cbe20c · merge 7cbe20c
 n/a provision failed #4218 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
23:25:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T23:55:10.585Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T23:55:12.507Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T23:55:13.792Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=dcd9608e40f4d3412620dde6abf1be2f6e66fb94a15c7a5937888afd3511bbf4
time=2026-03-04T23:56:44.857Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9404288/providers/Microsoft.Resources/deployments/dcd9608e40f4d3412620dde6abf1be2f6e66fb94a15c7a5937888afd3511bbf4/operationStatuses/08584289383724665115\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9404288/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9404288/providers/Microsoft.Monitor/accounts/services-j9404288/providers/Microsoft.Authorization/roleAssignments/b072200c-3836-5d7b-a0cc-8762024231bc\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T235534Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd3a4b710-b813-410b-822b-980dd1d8facc', request correlation id 'b27744b3-4c11-44fe-a8ae-a4f561e841f6'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 89abb52 · merge 89abb52
 n/a n/a passed #4299 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7cbe20c · merge 7cbe20c
 n/a n/a passed #4218 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c59e321 · merge 244d96c
 n/a n/a passed #4319 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
22:33:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T23:06:15.180Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T23:06:17.640Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T23:06:18.952Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=64274811a5ef1de0087200767591e57def13f2d84e6732be5258beb03413ef2f
time=2026-03-04T23:08:20.349Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5247744/providers/Microsoft.Resources/deployments/64274811a5ef1de0087200767591e57def13f2d84e6732be5258beb03413ef2f/operationStatuses/08584289413073485846\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5247744/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5247744/providers/Microsoft.Monitor/accounts/hcps-j5247744/providers/Microsoft.Authorization/roleAssignments/569de546-f40c-5a2d-9e32-a0e0f6144e5c\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T230649Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0388b196-e90d-44d2-8660-3f9a774dd473', request correlation id 'fb523f0a-3beb-48dc-9e26-032a91afc0d9'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
22:30:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T23:01:52.702Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T23:01:55.035Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T23:01:56.362Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=781e055fba988dc0f3fd638ee8d2dc2a1db61385c3a489642509e739bbc05ba3
time=2026-03-04T23:03:57.233Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9885056/providers/Microsoft.Resources/deployments/781e055fba988dc0f3fd638ee8d2dc2a1db61385c3a489642509e739bbc05ba3/operationStatuses/08584289415698775263\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9885056/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9885056/providers/Microsoft.Monitor/accounts/services-j9885056/providers/Microsoft.Authorization/roleAssignments/6f3d6dcb-1e92-56b7-93bb-abe94ff265e2\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T230248Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '1c621ad6-ca10-49c4-9430-cd0ec461ca86', request correlation id 'b666a097-93df-480c-bffc-f56e803edc29'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
22:27:20 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T22:54:10.500Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T22:54:12.028Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T22:54:13.442Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9c9e8a99c517a55b3fdc1edcd4c7c9781a8fb3bb0cfed133ecd1933c1e92b878
time=2026-03-04T22:56:44.608Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3551616/providers/Microsoft.Resources/deployments/9c9e8a99c517a55b3fdc1edcd4c7c9781a8fb3bb0cfed133ecd1933c1e92b878/operationStatuses/08584289420329641892\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3551616/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3551616/providers/Microsoft.Monitor/accounts/services-j3551616/providers/Microsoft.Authorization/roleAssignments/fd1c4b0e-e117-5355-9d18-185137e542d7\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T225627Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e0b25da4-fc54-4d66-a1fc-0a012dd7c572', request correlation id '64334af4-052b-42c8-a65c-a1a35ff3c555'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3551616/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3551616/providers/Microsoft.Monitor/accounts/hcps-j3551616/providers/Microsoft.Authorization/roleAssignments/63558cc7-12a9-53a5-8ad4-5ad66d246742\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T225628Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '46666170-7963-4ecd-be62-aa6eba00e8bb', request correlation id '64334af4-052b-42c8-a65c-a1a35ff3c555'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2f25940 · merge 8550ece
 n/a provision failed #4219 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
22:05:38 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T22:38:32.982Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T22:38:35.251Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T22:38:36.424Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=cb7a03da1653cb23813e75d6c2aaacde0530500ff8410cdafdfababf66e13a58
time=2026-03-04T22:41:37.548Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9100032/providers/Microsoft.Resources/deployments/cb7a03da1653cb23813e75d6c2aaacde0530500ff8410cdafdfababf66e13a58/operationStatuses/08584289429697883904\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9100032/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9100032/providers/Microsoft.Monitor/accounts/hcps-j9100032/providers/Microsoft.Authorization/roleAssignments/48d6559e-d792-586f-9fed-e819409a0132\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T224106Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '568ac856-0693-40e6-b9c3-dcbb5ffc3972', request correlation id 'f2f36624-97c0-4e15-9dce-61082732d936'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head db95ca2 · merge db95ca2
 n/a provision failed #4321 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
21:41:35 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T22:07:24.751Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T22:07:27.147Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T22:07:28.320Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=4f5ae9d00d4f03e57c42ea2fd4fded5c6ad0d5fd0fccbbaf869b48f9986dcf91
time=2026-03-04T22:09:29.365Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5805696/providers/Microsoft.Resources/deployments/4f5ae9d00d4f03e57c42ea2fd4fded5c6ad0d5fd0fccbbaf869b48f9986dcf91/operationStatuses/08584289448378848505\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5805696/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5805696/providers/Microsoft.Monitor/accounts/services-j5805696/providers/Microsoft.Authorization/roleAssignments/8de06612-55e5-5107-b071-362a6c69e1b8\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T220741Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '39bbf176-8363-489e-8417-42cabcb236e5', request correlation id '985c1e1f-f300-420f-ba2f-46fb722f2c02'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0381114
 n/a provision failed #4157 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
21:28:18 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T21:55:00.564Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T21:55:02.462Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T21:55:04.523Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=16534738d0ceec5d3ed4b7a0cc105ed7a31f056e999f0a7a110a876e07c59dd7
time=2026-03-04T21:56:35.408Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3215872/providers/Microsoft.Resources/deployments/16534738d0ceec5d3ed4b7a0cc105ed7a31f056e999f0a7a110a876e07c59dd7/operationStatuses/08584289455824981586\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3215872/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3215872/providers/Microsoft.Monitor/accounts/hcps-j3215872/providers/Microsoft.Authorization/roleAssignments/c3eea95f-2a47-598c-8c99-a4f0f6dad3e0\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T215525Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7d6b2bd0-20db-4f59-b3ae-227ca9b2df8e', request correlation id 'a4acfe60-8934-4643-b4d0-bc7bb3dfe921'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 9bce19c · merge efa8e74
 n/a n/a passed #4279 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7cbe20c · merge 7cbe20c
 n/a n/a passed #4218 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 9bce19c · merge efa8e74
 n/a provision failed #4279 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
20:39:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T21:07:20.042Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T21:07:21.971Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T21:07:23.065Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=e7aebd1a20481f960a46afad379b5c7398aaf21867cdf67fb3e0b428c9ad66ad
time=2026-03-04T21:08:53.586Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5193728/providers/Microsoft.Resources/deployments/e7aebd1a20481f960a46afad379b5c7398aaf21867cdf67fb3e0b428c9ad66ad/operationStatuses/08584289484430528698\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5193728/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5193728/providers/Microsoft.Monitor/accounts/hcps-j5193728/providers/Microsoft.Authorization/roleAssignments/2a533807-f55e-57bb-a22c-ea01a80ef8c5\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T210743Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '53f9b98c-3b27-46ce-9a62-576d11993cf9', request correlation id '12e72819-3dfc-4fb6-9140-cb44dab82bb5'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5193728/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5193728/providers/Microsoft.Monitor/accounts/services-j5193728/providers/Microsoft.Authorization/roleAssignments/5dab1394-c7a3-5866-9dc2-e6f5ed27dc2c\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T210753Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '15bcb1c0-5980-48aa-b69c-d227395e9a0b', request correlation id '12e72819-3dfc-4fb6-9140-cb44dab82bb5'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 013c940 · merge 0bc9efd
 n/a n/a passed #4320 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a n/a passed #4243 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
19:55:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T20:28:33.469Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T20:28:35.632Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T20:28:36.743Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=1e2746236a223cc82826d40c7a221258c815eaa771d177b618b6eedc14af321e
time=2026-03-04T20:30:37.678Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8689920/providers/Microsoft.Resources/deployments/1e2746236a223cc82826d40c7a221258c815eaa771d177b618b6eedc14af321e/operationStatuses/08584289507694457557\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8689920/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8689920/providers/Microsoft.Monitor/accounts/services-j8689920/providers/Microsoft.Authorization/roleAssignments/6d179d95-0fb6-5120-81e3-0bbfe064445a\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T202852Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '917ac4ea-b671-4207-bb9e-ff38aab9941d', request correlation id '3fdbb3ee-b775-4232-8b7d-49154b8b744f'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a n/a passed #4243 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head f879b10 · merge f879b10
 n/a n/a passed #4289 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b60ce1f · merge 138964d
 n/a e2e failed ⚠#4226 (merged) 1
e2e (1)
expected 2 nodes to still have original taint key1=value1:NoSchedule
19:46:43 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:232]: expected 2 nodes to still have original taint key1=value1:NoSchedule
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:232]: expected 2 nodes to still have original taint key1=value1:NoSchedule
Expected
    <bool>: false
to be true
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b61beb9 · merge b61beb9
 n/a n/a passed #4285 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head afb6057
 n/a e2e failed ⚠#4198 (open) 11
e2e (11)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
19:33:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a0f800>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-ds7hg6, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-ds7hg6" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-ds7hg6, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ds7hg6\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000784f80>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-ds7hg6",
            },
            <*fmt.wrapError | 0xc00017b2c0>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ds7hg6\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a0f800>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-ds7hg6, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-ds7hg6" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-ds7hg6, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ds7hg6\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000784f80>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-ds7hg6",
            },
            <*fmt.wrapError | 0xc00017b2c0>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ds7hg6\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
19:33:47 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:242]: Unexpected error:
    <*fmt.wrapErrors | 0xc000b92c00>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-hx4qt7, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hx4qt7" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-hx4qt7, error: failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-hx4qt7\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000653b60>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-hx4qt7",
            },
            <*fmt.wrapError | 0xc0003c8fc0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-hx4qt7\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:242]: Unexpected error:
    <*fmt.wrapErrors | 0xc000b92c00>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-hx4qt7, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hx4qt7" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-hx4qt7, error: failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-hx4qt7\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000653b60>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-hx4qt7",
            },
            <*fmt.wrapError | 0xc0003c8fc0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-hx4qt7\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
19:33:47 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc000efb0b0>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-n9h7f8, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-n9h7f8" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-n9h7f8, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-n9h7f8\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012c5340>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-n9h7f8",
            },
            <*fmt.wrapError | 0xc00080ab00>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-n9h7f8\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc000efb0b0>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-n9h7f8, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-n9h7f8" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-n9h7f8, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-n9h7f8\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012c5340>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-n9h7f8",
            },
            <*fmt.wrapError | 0xc00080ab00>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-n9h7f8\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift
19:33:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapError | 0xc000b8caa0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-ws5kq6": PUT https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-ws5kq6\": PUT https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000b8ca80>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-ws5kq6\": PUT https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010f5650>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["345"],
                        "X-Envoy-Upstream-Service-Time": ["76"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "9eda12b4-bb47-465d-8806-c9ac50df5fa4",
                        ],
                        "Date": [
                            "Wed, 04 Mar 2026 21:03:05 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000b8c9a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapError | 0xc000b8caa0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-ws5kq6": PUT https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-ws5kq6\": PUT https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000b8ca80>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-ws5kq6\": PUT https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws5kq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010f5650>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["345"],
                        "X-Envoy-Upstream-Service-Time": ["76"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "9eda12b4-bb47-465d-8806-c9ac50df5fa4",
                        ],
                        "Date": [
                            "Wed, 04 Mar 2026 21:03:05 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000b8c9a0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
19:33:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc00146c060>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-4xdqrt, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-4xdqrt" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-4xdqrt, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4xdqrt\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0005b23c0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-4xdqrt",
            },
            <*fmt.wrapError | 0xc000cdc020>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4xdqrt\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc00146c060>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-4xdqrt, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-4xdqrt" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-4xdqrt, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4xdqrt\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0005b23c0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-4xdqrt",
            },
            <*fmt.wrapError | 0xc000cdc020>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4xdqrt\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:33:47 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000850408>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list96w5f7" in resourcegroup="ea-list-hhrk4k" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009ee640>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list96w5f7\" in resourcegroup=\"ea-list-hhrk4k\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009ee620>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list96w5f7\" in resourcegroup=\"ea-list-hhrk4k\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009ee600>{
                        msg: "failed waiting for hcpCluster=\"ea-list96w5f7\" in resourcegroup=\"ea-list-hhrk4k\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0260b0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000850408>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list96w5f7" in resourcegroup="ea-list-hhrk4k" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009ee640>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list96w5f7\" in resourcegroup=\"ea-list-hhrk4k\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009ee620>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list96w5f7\" in resourcegroup=\"ea-list-hhrk4k\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009ee600>{
                        msg: "failed waiting for hcpCluster=\"ea-list96w5f7\" in resourcegroup=\"ea-list-hhrk4k\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0260b0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
19:33:47 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e66030>: 
    failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-blmsns" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-blmsns, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-blmsns\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-blmsns, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014c5270>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-blmsns",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e66030>: 
    failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-blmsns" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-blmsns, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-blmsns\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-blmsns, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014c5270>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-blmsns",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift
19:33:47 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:92]: Unexpected error:
    <*fmt.wrapError | 0xc000a08d00>: 
    failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0011615c0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "39a12661-31ec-4bb4-9474-f1a1fecdcb20",
                    ],
                    "Date": [
                        "Wed, 04 Mar 2026 21:03:38 GMT",
                    ],
                    "Content-Length": ["336"],
                    "X-Envoy-Upstream-Service-Time": ["24"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000a08c20>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 110, 111, 100, 101, 112, 111, 111, 108, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 102, 119, 113, 56, 107, 102, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 110, 112, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 336,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j7517312.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/974ebd46-8ad3-41e3-afef-7...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:92]: Unexpected error:
    <*fmt.wrapError | 0xc000a08d00>: 
    failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-fwq8kf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0011615c0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "39a12661-31ec-4bb4-9474-f1a1fecdcb20",
                    ],
                    "Date": [
                        "Wed, 04 Mar 2026 21:03:38 GMT",
                    ],
                    "Content-Length": ["336"],
                    "X-Envoy-Upstream-Service-Time": ["24"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000a08c20>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 110, 111, 100, 101, 112, 111, 111, 108, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 102, 119, 113, 56, 107, 102, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 110, 112, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 336,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j7517312.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/974ebd46-8ad3-41e3-afef-7...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
19:33:47 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: Unexpected error:
    <*fmt.wrapErrors | 0xc001482e40>: 
    failed to create NodePool autoscale-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-np in resource group np-autoscaling-52w4cj, error: failed waiting for nodepool="autoscale-np" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-52w4cj" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool autoscale-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-np in resource group np-autoscaling-52w4cj, error: failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52w4cj\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001101540>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-np in resource group np-autoscaling-52w4cj",
            },
            <*fmt.wrapError | 0xc00150a580>{
                msg: "failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52w4cj\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: Unexpected error:
    <*fmt.wrapErrors | 0xc001482e40>: 
    failed to create NodePool autoscale-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-np in resource group np-autoscaling-52w4cj, error: failed waiting for nodepool="autoscale-np" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-52w4cj" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool autoscale-np, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-np in resource group np-autoscaling-52w4cj, error: failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52w4cj\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001101540>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-np in resource group np-autoscaling-52w4cj",
            },
            <*fmt.wrapError | 0xc00150a580>{
                msg: "failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52w4cj\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
ERROR CODE: Conflict; detail message Cannot request credential while resource is "accepted"; provider Microsoft.RedHatOpenShift
19:33:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:156]: Unexpected error:
    <*fmt.wrapError | 0xc0009807c0>: 
    failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0015252f0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "dc4fe3fc-5713-4a7a-848c-01bfb109e432",
                    ],
                    "Date": [
                        "Wed, 04 Mar 2026 21:19:21 GMT",
                    ],
                    "Content-Length": ["323"],
                    "X-Envoy-Upstream-Service-Time": ["27"],
                    "Server": ["istio-envoy"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009806c0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 97, 99, 99, 101, 112, 116, 101, 100, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 114, 103, 45, 99, 108, 117, 115, 116, 101, 114, 45, 98, 97, 99, 107, 45, 118, 101, 114, 115, 105, 111, 110, 45, 55, 98, 112, 55, 114, 57, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 99, 108, 117, 115, 116, 101, 114, 45, 118, 101, 114, 45, 52, 45, 49, 57, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 323,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j7517312.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcp...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:156]: Unexpected error:
    <*fmt.wrapError | 0xc0009807c0>: 
    failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j7517312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0015252f0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "dc4fe3fc-5713-4a7a-848c-01bfb109e432",
                    ],
                    "Date": [
                        "Wed, 04 Mar 2026 21:19:21 GMT",
                    ],
                    "Content-Length": ["323"],
                    "X-Envoy-Upstream-Service-Time": ["27"],
                    "Server": ["istio-envoy"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009806c0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 97, 99, 99, 101, 112, 116, 101, 100, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 114, 103, 45, 99, 108, 117, 115, 116, 101, 114, 45, 98, 97, 99, 107, 45, 118, 101, 114, 115, 105, 111, 110, 45, 55, 98, 112, 55, 114, 57, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 99, 108, 117, 115, 116, 101, 114, 45, 118, 101, 114, 45, 52, 45, 49, 57, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 323,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j7517312.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-7bp7r9/providers/Microsoft.RedHatOpenShift/hcp...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
19:33:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc001678de0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-7779tn, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7779tn" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-7779tn, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7779tn\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00176feb0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-7779tn",
            },
            <*fmt.wrapError | 0xc0004ec8c0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7779tn\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc001678de0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-7779tn, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7779tn" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-7779tn, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7779tn\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00176feb0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-7779tn",
            },
            <*fmt.wrapError | 0xc0004ec8c0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7779tn\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head efd28c6 · merge efd28c6
 n/a provision failed #4284 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
19:29:47 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T20:03:27.202Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T20:03:29.325Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T20:03:30.674Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=114589f546f36207463d98126d188d1970f8c06b6e43e7fa9056fbaea567b080
time=2026-03-04T20:05:31.658Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8942720/providers/Microsoft.Resources/deployments/114589f546f36207463d98126d188d1970f8c06b6e43e7fa9056fbaea567b080/operationStatuses/08584289522756264799\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8942720/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8942720/providers/Microsoft.Monitor/accounts/services-j8942720/providers/Microsoft.Authorization/roleAssignments/fc4772bd-3e5c-5b5d-a99a-a56d1dcc3d3f\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T200344Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'fd1bb717-3e55-473a-9f0c-063e115fc731', request correlation id '50eabdd7-ca4b-4468-a1ad-353af33ce524'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8942720/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8942720/providers/Microsoft.Monitor/accounts/hcps-j8942720/providers/Microsoft.Authorization/roleAssignments/ec8367bb-50a2-5149-b952-50f32d1ef6a8\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T200352Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'ed6d8a05-f41f-4c70-866a-a57600402054', request correlation id '50eabdd7-ca4b-4468-a1ad-353af33ce524'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c59e321 · merge 244d96c
 n/a provision failed #4319 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
19:28:24 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T19:58:35.089Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T19:58:37.792Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T19:58:39.149Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=6a16024d1e9f409c92033f499d2defc00d92c0dac6a1acd92f07ad03ec094979
time=2026-03-04T20:00:40.282Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2628864/providers/Microsoft.Resources/deployments/6a16024d1e9f409c92033f499d2defc00d92c0dac6a1acd92f07ad03ec094979/operationStatuses/08584289525671888074\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2628864/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2628864/providers/Microsoft.Monitor/accounts/services-j2628864/providers/Microsoft.Authorization/roleAssignments/8b284b8a-20cf-5023-b694-cb3fbe719ae6\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T195854Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0fe8dd9e-695e-43e9-8a26-426c7e5d97e3', request correlation id 'df8e2676-c454-4096-9185-ec65a8b0ab6e'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
18:48:10 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T19:18:07.259Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T19:18:09.297Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T19:18:10.883Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=1d977f7d908bb0d6fe0e7d76ab760a2d4bb50f8189660bd1895b23b0772b8a4c
time=2026-03-04T19:20:11.968Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5857920/providers/Microsoft.Resources/deployments/1d977f7d908bb0d6fe0e7d76ab760a2d4bb50f8189660bd1895b23b0772b8a4c/operationStatuses/08584289549956731661\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5857920/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5857920/providers/Microsoft.Monitor/accounts/hcps-j5857920/providers/Microsoft.Authorization/roleAssignments/18509be4-9027-57b5-8639-cbaf540b3286\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T191831Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '90c10169-f367-485e-94d9-aa30eb3ef32b', request correlation id '929bfc43-4321-4b7d-8d07-5f84a153792b'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7148a34 · merge 7148a34
 n/a n/a passed #4303 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4b5ae23 · merge 4b5ae23
 n/a provision failed #4310 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
17:50:27 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T18:35:02.105Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T18:35:04.165Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T18:35:05.424Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=ad1d26434c191aa2798b467a2b633dc3445d6b35562229560a1c41b231bc6af6
time=2026-03-04T18:37:06.921Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7933696/providers/Microsoft.Resources/deployments/ad1d26434c191aa2798b467a2b633dc3445d6b35562229560a1c41b231bc6af6/operationStatuses/08584289575808313886\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7933696/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7933696/providers/Microsoft.Monitor/accounts/services-j7933696/providers/Microsoft.Authorization/roleAssignments/5ee749ef-c3ec-5b35-b998-da7137950502\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T183615Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'c3b68396-ddec-46cd-b39a-9b13ceb9a629', request correlation id '4f15cad2-9417-4640-a9e3-8ee022b1de7e'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0b3a455 · merge 0b3a455
 n/a n/a passed #4315 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 35511ce · merge 35511ce
 n/a provision failed #4314 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
17:16:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T17:49:05.546Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T17:49:08.081Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T17:49:09.262Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=8e060c96442b359ddd0d3602923fedefc46092bc3a046ecc9e71e02f11d53b75
time=2026-03-04T17:51:10.126Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4764416/providers/Microsoft.Resources/deployments/8e060c96442b359ddd0d3602923fedefc46092bc3a046ecc9e71e02f11d53b75/operationStatuses/08584289603369475030\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4764416/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4764416/providers/Microsoft.Monitor/accounts/services-j4764416/providers/Microsoft.Authorization/roleAssignments/51e70bce-e48e-5148-b2c5-6ac918d25157\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T174924Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82ff6217-fc88-488c-a5bc-e72aa66bab19', request correlation id '1676f6e7-e03a-4094-94ea-ef92ca94cc46'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4764416/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4764416/providers/Microsoft.Monitor/accounts/hcps-j4764416/providers/Microsoft.Authorization/roleAssignments/939d22ac-0fcd-581c-a376-a04f2a0017ff\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T174921Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5f75a6f5-273a-469e-af24-e6d151d5f962', request correlation id '1676f6e7-e03a-4094-94ea-ef92ca94cc46'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 52c9c69 · merge 0ac5e19
 n/a provision failed #4181 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
16:40:34 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T17:12:01.264Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T17:12:03.132Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T17:12:04.363Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=4367a7d3e2f3e375f64a27aa555715150c21e58b0224619e5ab6f81e59b76d47
time=2026-03-04T17:13:35.222Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7816960/providers/Microsoft.Resources/deployments/4367a7d3e2f3e375f64a27aa555715150c21e58b0224619e5ab6f81e59b76d47/operationStatuses/08584289625617683621\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7816960/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7816960/providers/Microsoft.Monitor/accounts/hcps-j7816960/providers/Microsoft.Authorization/roleAssignments/2d6818b1-8d22-58e9-8a48-c9a74b129989\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T171219Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9a1c3e98-ed1f-4270-b1ea-1f180744dbf9', request correlation id 'beeaf399-2261-4d54-ab5f-3e86f1af80bb'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7816960/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7816960/providers/Microsoft.Monitor/accounts/services-j7816960/providers/Microsoft.Authorization/roleAssignments/8b526356-7b87-5bf7-9086-1924759a5eee\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T171228Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '24e5a517-95bb-4729-8007-d3e1e7a5b105', request correlation id 'beeaf399-2261-4d54-ab5f-3e86f1af80bb'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b60ce1f · merge 138964d
 n/a provision failed #4226 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
16:34:35 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T17:15:18.278Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T17:15:20.533Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T17:15:21.781Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d715d7f2f86c8d9c8a18d69fd57d4c08c3094631b0ab9e92abd7ce843db5585d
time=2026-03-04T17:16:52.478Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0741632/providers/Microsoft.Resources/deployments/d715d7f2f86c8d9c8a18d69fd57d4c08c3094631b0ab9e92abd7ce843db5585d/operationStatuses/08584289623644387667\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0741632/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0741632/providers/Microsoft.Monitor/accounts/services-j0741632/providers/Microsoft.Authorization/roleAssignments/e0314240-293c-5d24-b673-615952444f06\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T171537Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'bceb4411-ee20-4276-9faa-436d38bb2a38', request correlation id '8fdb6523-a3ec-47db-9e20-290098faf894'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0741632/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0741632/providers/Microsoft.Monitor/accounts/hcps-j0741632/providers/Microsoft.Authorization/roleAssignments/722ecb59-4670-531e-8c1d-3d1f218cad3f\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T171613Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '896a8848-22a3-4ec8-9ac9-1a25af81a7da', request correlation id '8fdb6523-a3ec-47db-9e20-290098faf894'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6e48d06 · merge 6e48d06
 n/a n/a passed #4318 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a59797e · merge 4c15703
 n/a provision failed #4075 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
16:29:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T17:03:57.931Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T17:04:00.330Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T17:04:01.659Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9c599b7af13588fd58ca5a46b5550cc3c3631f025471032cd316fd5e7b184c20
time=2026-03-04T17:05:32.407Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6371968/providers/Microsoft.Resources/deployments/9c599b7af13588fd58ca5a46b5550cc3c3631f025471032cd316fd5e7b184c20/operationStatuses/08584289630446024430\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6371968/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6371968/providers/Microsoft.Monitor/accounts/services-j6371968/providers/Microsoft.Authorization/roleAssignments/64c6e2a3-0acf-5ef1-b40a-cfe8e721ec65\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T170414Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3ccc6d38-1264-4093-9f38-b4e19ded1fd4', request correlation id '51d6ed71-fac7-4caa-9d47-d4eb5b2c8863'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6371968/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6371968/providers/Microsoft.Monitor/accounts/hcps-j6371968/providers/Microsoft.Authorization/roleAssignments/18358df3-69f2-5b96-8489-24a2425bf3c5\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T170416Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'adb321d9-6ff6-46a1-9550-baabb063e081', request correlation id '51d6ed71-fac7-4caa-9d47-d4eb5b2c8863'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2bbe0eb · merge 4c15703
 n/a unknown failed #4075 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a n/a passed #4243 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head bc6c374 · merge 3832c3f
 n/a provision failed ⚠#4317 (merged) 1
provision (1)
resource not ready, name: maestro-db, kind: Deployment, status: Failed
15:45:21 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Server/service/deploy · suite step graph
Show raw failure
time=2026-03-04T16:42:54.462Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-04T16:42:55.072Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:52268a48-3ba3-4eb7-bae0-330881d68626 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3604096.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3604096] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:3 requests:map[cpu:200m memory:512Mi] zoneCount:4] image:map[digest:sha256:144f3abdc7e843bf8a59b3b45c9892528e22de6068567fb6cfd26630f33f621e registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:062f0656-796b-42b2-99c8-7ecf6461937b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T16:42:55.072Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:42:55.379Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-03-04T16:42:55.788Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-03-04T16:42:55.788Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:42:56.022Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:48:03.838Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T16:48:03.838Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:48:03.838Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:48:08.035Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy release=maestro-server namespace=maestro status=failed description="Release \"maestro-server\" failed: resource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[clientId:52268a48-3ba3-4eb7-bae0-330881d68626 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3604096.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3604096] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:3 requests:map[cpu:200m memory:512Mi] zoneCount:4] image:map[digest:sha256:144f3abdc7e843bf8a59b3b45c9892528e22de6068567fb6cfd26630f33f621e registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:062f0656-796b-42b2-99c8-7ecf6461937b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T16:48:08.037Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yUQW/TQBCF7/4VI58SKa5CaKsqqIdSOCFQJBCHIg7r3Ye9yXrXmh0nCsqPR04oMa3TGG7r2e+NZ9+MJnEQYsTQsEakWzJKlKjcYRTyJbR8sN7Mo7D1xYQOoU+qwp8Q1vD7SKyVfgyPvyVEROln8Npq3GkdGi/phNJKIQqHv44HdgGONgq8fA2uqXDvlK06XGbyHtXvP3S5gms9jCyhnJS6hF4NEgzLWkHY6jgM7n3UO9QubCucs6yXG5KyDlEKRsxyhloVTsW+cu80Lxrn3lpvrC/2usa5LD9+PxU0Uga2P5XY4BfBWb1tKeVc2GS50it4868y7Zoo4Cz2dGOA/NEUOeHLy+KTnXxR5oOU/2EQo7BReH+XLUPe1nx28D4GbyVwt/+tU+BeiWacGalnTL9tC4Dbx8CL1fuKW8jgh2qcnEy74LC2Bnz/bOCOx+T7m2TV5GAPQXzfbpeY7GhTgkFiK0RRVU05ZAN4GhklaMOjdDadXWfT19n08sur6/nlbH519ZCO6eKCTjI38+nNQzoeJztaButpZb25td6DOzsxeDquwu4OfLr8kh3VHNrbY6ETqoM5wMOSTMiFYkIVYlRFmzKwAVO+7TzeIOpfAQAA//+ztJdiuQUAAA=="
time=2026-03-04T16:48:08.040Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:48:08.041Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T16:50:08.042Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-04T16:50:09.026Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:52268a48-3ba3-4eb7-bae0-330881d68626 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3604096.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3604096] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:3 requests:map[cpu:200m memory:512Mi] zoneCount:4] image:map[digest:sha256:144f3abdc7e843bf8a59b3b45c9892528e22de6068567fb6cfd26630f33f621e registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:062f0656-796b-42b2-99c8-7ecf6461937b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T16:50:09.026Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:50:09.534Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-03-04T16:50:09.802Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:50:09.802Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:50:10.016Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy numVersions=1
time=2026-03-04T16:50:11.026Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:50:11.026Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=ServiceAccount" namespace=maestro name=maestro
time=2026-03-04T16:50:11.399Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=PersistentVolumeClaim" namespace=maestro name=maestro-db
time=2026-03-04T16:50:11.762Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-grpc
time=2026-03-04T16:50:12.092Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-healthcheck
time=2026-03-04T16:50:12.212Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro
time=2026-03-04T16:50:12.330Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-metrics
time=2026-03-04T16:50:12.451Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-db
time=2026-03-04T16:50:12.572Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=maestro name=maestro
time=2026-03-04T16:50:12.918Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=maestro name=maestro-db
time=2026-03-04T16:50:13.038Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=maestro name=postgres-breakglass
time=2026-03-04T16:50:13.158Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="acrpull.microsoft.com/v1beta2, Kind=AcrPullBinding" namespace=maestro name=pull-binding
time=2026-03-04T16:50:13.231Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=72.209865ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/acrpull.microsoft.com/v1beta2/namespaces/maestro/acrpullbindings/pull-binding
time=2026-03-04T16:50:13.431Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=139.293289ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/acrpull.microsoft.com/v1beta2/namespaces/maestro/acrpullbindings/pull-binding
time=2026-03-04T16:50:13.697Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-backend
time=2026-03-04T16:50:13.831Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=74.208758ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-backend
time=2026-03-04T16:50:14.148Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-cluster-service
time=2026-03-04T16:50:14.349Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-maestro-to-db
time=2026-03-04T16:50:14.431Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=81.237342ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-maestro-to-db
time=2026-03-04T16:50:14.631Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=140.336941ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-maestro-to-db
time=2026-03-04T16:50:14.761Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-metrics
time=2026-03-04T16:50:14.831Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=70.238372ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-metrics
time=2026-03-04T16:50:15.031Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=141.282922ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-metrics
time=2026-03-04T16:50:15.106Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-nothing
time=2026-03-04T16:50:15.231Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=125.220112ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-nothing
time=2026-03-04T16:50:15.431Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=141.331893ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-nothing
time=2026-03-04T16:50:15.496Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-registration-job-to-maestro
time=2026-03-04T16:50:15.630Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=134.230097ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-registration-job-to-maestro
time=2026-03-04T16:50:15.831Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=140.27005ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-registration-job-to-maestro
time=2026-03-04T16:50:15.899Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1beta1, Kind=PeerAuthentication" namespace=maestro name=default
time=2026-03-04T16:50:16.030Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=131.260133ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1beta1/namespaces/maestro/peerauthentications/default
time=2026-03-04T16:50:16.230Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=137.247314ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1beta1/namespaces/maestro/peerauthentications/default
time=2026-03-04T16:50:16.489Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="secrets-store.csi.x-k8s.io/v1, Kind=SecretProviderClass" namespace=maestro name=maestro
time=2026-03-04T16:50:16.631Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=82.198643ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/secrets-store.csi.x-k8s.io/v1/namespaces/maestro/secretproviderclasses/maestro
time=2026-03-04T16:50:16.901Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=maestro name=maestro-server
time=2026-03-04T16:50:17.030Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=66.138743ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/monitoring.coreos.com/v1/namespaces/maestro/servicemonitors/maestro-server
time=2026-03-04T16:50:17.315Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-03-04T16:50:17.315Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:50:17.482Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy numVersions=1
time=2026-03-04T16:55:21.909Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T16:55:21.909Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:55:21.909Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:55:23.755Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy release=maestro-server namespace=maestro status=failed description="Upgrade \"maestro-server\" failed: resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded" values="map[azure:map[clientId:52268a48-3ba3-4eb7-bae0-330881d68626 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3604096.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3604096] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:3 requests:map[cpu:200m memory:512Mi] zoneCount:4] image:map[digest:sha256:144f3abdc7e843bf8a59b3b45c9892528e22de6068567fb6cfd26630f33f621e registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:062f0656-796b-42b2-99c8-7ecf6461937b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T16:55:23.757Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5SUQWvbMBTH7/4UwqcE7JKmawcZPXTdTmMjsLFDxw6y9J+tRNYzT88JGfnww0nbZK1DnaOffr8/T7L0Eg9RjEgtG0R1q6wWLbrwGFGxgJEvLthZFHahzNS+9E3XeC5hhbCrxEabp/L4V6KUUuldKxWx+6vFUZiTd2aTZirV3tM6L7RZItiuUGtEYUqzAZrxbRRwHsErZ3Cu/sjmQrktzpYh7Ew8VwsklQvluRqjdFF4t5YvqOh6fvZfRc0B7uIQxJmd00EWf3TrpYf/vj+/rxScEB8Bu5MFn1bujKE2/J/ZwxrGuUz/L5mDo4uCID/JtzXuvXb1m9YnNJ42Nd5qtJcbEtlQlJIR84Khl6XXse9i7Pc4Z1o5C75/RZ065ONmSm7MMLKC9lKZCmY5SBiWevra98D9r8rwvPX+owv28SE0rfd5cfh+EpLfH5JlW4ADBPFzN11islXrCgwlrkYUXTeqgKyBoEZWC7ryKJ1Opjf55CqfvPtxeTO7nswu3z+kY3VxoU4y17Pp1UM6HidbtSAX1NIFe+tCAB/NRArqMAqPZ+DL4ZdsVcPUrR4azVRDdg8PC8mUpzJTNWLUZRdJbMGq2Bxt3iKafwEAAP//n0DhhrkFAAA="
time=2026-03-04T16:55:23.761Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:55:23.761Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T16:57:23.762Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-04T16:57:24.576Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:52268a48-3ba3-4eb7-bae0-330881d68626 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3604096.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3604096] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:3 requests:map[cpu:200m memory:512Mi] zoneCount:4] image:map[digest:sha256:144f3abdc7e843bf8a59b3b45c9892528e22de6068567fb6cfd26630f33f621e registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:062f0656-796b-42b2-99c8-7ecf6461937b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T16:57:24.576Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:57:25.037Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-03-04T16:57:25.540Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:57:25.540Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:57:25.801Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy numVersions=2
time=2026-03-04T16:57:26.867Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:57:26.867Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=ServiceAccount" namespace=maestro name=maestro
time=2026-03-04T16:57:27.471Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=PersistentVolumeClaim" namespace=maestro name=maestro-db
time=2026-03-04T16:57:27.984Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-grpc
time=2026-03-04T16:57:28.512Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-healthcheck
time=2026-03-04T16:57:28.632Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro
time=2026-03-04T16:57:28.756Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-metrics
time=2026-03-04T16:57:28.876Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=maestro name=maestro-db
time=2026-03-04T16:57:28.997Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=maestro name=maestro
time=2026-03-04T16:57:29.528Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=maestro name=maestro-db
time=2026-03-04T16:57:29.659Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=maestro name=postgres-breakglass
time=2026-03-04T16:57:29.783Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="acrpull.microsoft.com/v1beta2, Kind=AcrPullBinding" namespace=maestro name=pull-binding
time=2026-03-04T16:57:30.288Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-backend
time=2026-03-04T16:57:30.837Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-cluster-service
time=2026-03-04T16:57:31.015Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-maestro-to-db
time=2026-03-04T16:57:31.220Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-metrics
time=2026-03-04T16:57:31.380Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-nothing
time=2026-03-04T16:57:31.514Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=133.260993ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-nothing
time=2026-03-04T16:57:31.713Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=140.193665ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-nothing
time=2026-03-04T16:57:31.777Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=maestro name=allow-registration-job-to-maestro
time=2026-03-04T16:57:31.913Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=135.165595ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-registration-job-to-maestro
time=2026-03-04T16:57:32.113Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=139.121622ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1/namespaces/maestro/authorizationpolicies/allow-registration-job-to-maestro
time=2026-03-04T16:57:32.180Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="security.istio.io/v1beta1, Kind=PeerAuthentication" namespace=maestro name=default
time=2026-03-04T16:57:32.313Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy delay=133.271733ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j3604096-svc-u5wowc8e.hcp.westus3.azmk8s.io:443/apis/security.istio.io/v1beta1/namespaces/maestro/peerauthentications/default
time=2026-03-04T16:57:32.826Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="secrets-store.csi.x-k8s.io/v1, Kind=SecretProviderClass" namespace=maestro name=maestro
time=2026-03-04T16:57:33.282Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=maestro name=maestro-server
time=2026-03-04T16:57:33.780Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-03-04T16:57:33.780Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T16:57:33.949Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy numVersions=2
time=2026-03-04T17:02:38.179Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T17:02:38.179Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T17:02:38.179Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T17:02:42.674Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy release=maestro-server namespace=maestro status=failed description="Upgrade \"maestro-server\" failed: resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded" values="map[azure:map[clientId:52268a48-3ba3-4eb7-bae0-330881d68626 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3604096.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3604096] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:3 requests:map[cpu:200m memory:512Mi] zoneCount:4] image:map[digest:sha256:144f3abdc7e843bf8a59b3b45c9892528e22de6068567fb6cfd26630f33f621e registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:062f0656-796b-42b2-99c8-7ecf6461937b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T17:02:42.676Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5xUwW7aQBS8+yuefAoSjiikiUSVQ5r2VLVCouohVQ/r3am9sN613j6DqPj4yqAUQkzi9Ga/nRnvm7EmcRBixNCwRqRbMkqUqNzhIuQLaPlivZlGYeuLIe1H31SFfyOs4HeTWCv9OB78TIiI0jl4ZTXutA6Nl3RIaaUQhcOTxz12Bo42Crz8CK6pcO+UrY5wmck7WJ9Qu7Cp8Jp6J66PZB2iFIyY5Qy1LJyKsYNzp3nWOPfRemN9seM1zmX54f2U0EgZ2P5RYoOfBWf1pkUp58I6y5Vewpu30rRrooCzuHf9rfRHU+SMLy+TIWx1pzUv0XyQ8j8MYhQ2Cu/OskXI2zufz34OzZAZh5U14PtnEXYQdgZ+Dd5K4OMfprUWfPYbr8meYLp9nj+LLyu41v2QJZSTUpfQy16Efqrn0+0Ady41A7iNFF6s3uXWggx+q8Y9sS359SFZNjnYQxA/t+0Sky2tSzBIbIUoqqoph6wBTxdGCdrxRToeja+z0SQbXX1/dz19fzOdTB7SAV1eUjfmZjoaT6/GD+lgkGxpEaynpfXm1noPPurE4OlQhccdeFp+yZZqDu3p4aJDqoPZg/uJDMmFYkgVYlRFKxnYgCnfHC1vEPXfAAAA//+FAnesuQUAAA=="
time=2026-03-04T17:02:42.678Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-04T17:02:42.679Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T17:04:42.679Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro, kind: Deployment, status: Failed\nresource not ready, name: maestro-db, kind: Deployment, status: Failed\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b60ce1f · merge 138964d
 n/a provision failed #4226 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
15:17:35 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T15:18:46.947Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T15:18:49.217Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T15:18:50.746Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d498fef569e9f8d26aae977c9ab8a615395f9512dd21737f12a5032062dc88f8
time=2026-03-04T15:20:51.729Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4767360/providers/Microsoft.Resources/deployments/d498fef569e9f8d26aae977c9ab8a615395f9512dd21737f12a5032062dc88f8/operationStatuses/08584289693556557198\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4767360/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4767360/providers/Microsoft.Monitor/accounts/hcps-j4767360/providers/Microsoft.Authorization/roleAssignments/af63ca07-014b-555e-9683-9c45578cfd0c\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T151923Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'bcdc0dbb-84ee-42d8-8e8f-982ca44294f7', request correlation id '44d55cbe-c178-4702-84b1-23d37bc961db'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 35511ce · merge 35511ce
 n/a n/a passed #4314 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 5add497 · merge 8550ece
 n/a provision failed #4219 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
15:11:25 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T15:43:44.365Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T15:43:46.577Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T15:43:47.701Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=ece3ee87dbeed2435e682c0664a8a2bc4d71b6fbd1ca26ce3ec380e7399db59c
time=2026-03-04T15:45:18.541Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4679168/providers/Microsoft.Resources/deployments/ece3ee87dbeed2435e682c0664a8a2bc4d71b6fbd1ca26ce3ec380e7399db59c/operationStatuses/08584289678584280771\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4679168/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4679168/providers/Microsoft.Monitor/accounts/hcps-j4679168/providers/Microsoft.Authorization/roleAssignments/f48158a1-781a-5e71-931d-510e32066f5b\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T154404Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '45b42b6c-b21a-4396-8ee9-9ed0e9f95a88', request correlation id '6fe01212-66ae-426e-8fe0-a55d49d614b3'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 87ae44a · merge 87ae44a
 n/a n/a passed #4154 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0b3a455 · merge 0b3a455
 n/a n/a passed #4315 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 27be21c · merge a88a31f
 n/a provision failed #4316 (merged) 1
provision (1)
resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed context deadline exceeded
14:51:25 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy · suite step graph
Show raw failure
time=2026-03-04T15:40:58.991Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-04T15:40:59.568Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj2610048.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj2610048.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d databaseName:arohcpprow-rp-j2610048 databaseUrl:https://arohcpprow-rp-j2610048.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-1n42z7iz/pipeline@sha256:c0b52f4451b052ec0e26f17df6c414a1b8bd50fdf71ec8bed58bd55d2084ca5b replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-1n42z7iz/pipeline:pull workloadIdentityClientId:8b702cd7-2c4a-4e01-8dbe-6220f9c6da7f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T15:40:59.569Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:40:59.827Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-04T15:41:00.458Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-03-04T15:41:00.458Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:41:00.576Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:46:07.733Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T15:46:07.733Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:46:07.733Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:46:11.060Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj2610048.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj2610048.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d databaseName:arohcpprow-rp-j2610048 databaseUrl:https://arohcpprow-rp-j2610048.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-1n42z7iz/pipeline@sha256:c0b52f4451b052ec0e26f17df6c414a1b8bd50fdf71ec8bed58bd55d2084ca5b replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-1n42z7iz/pipeline:pull workloadIdentityClientId:8b702cd7-2c4a-4e01-8dbe-6220f9c6da7f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T15:46:11.061Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4xTwW4aMRC971eM9hSkdQRpmgNVDmnaU5UKNT2l6sFrv4KD17Mae0FEfHxlKGWFlpbrm/fG855nCo9EgsidGES6J6uTTrr2uOL6FSZ9ccFOYxIX5hXtoa+6wV8IK4QdElttDvDoR0FEVH5jj48uWBfmZUVlrc0SwSoPbSEKHiY5DrmkhdXCtGW1Fz5DVs6gV1EHcYMkzsQB0SOHX27+pNv+W/qtEyjpQnINlNlRLtWeZWdfGf6/kU9oPW8ahDTgZYD/YGTWeT8QWtt5r+ojfirs0oLFvek8yIy9M5sdy3teHzL7x7szQHILhOSMPjFzQfbPMII0E145C3n0Osb+8HH/n2qJjVrpzqfzf/7EwSWWy+L6o3kwhrt9xAPc4ueHYtnVkICE+DkvbCy2tF5AQHktYtJNSzXSGgh0ZXVChq/Km/HNnRq/U+Pb75P309vJdDx+KUd0fU1nOXfTyeSlHI2KLb2yC7R0wd67ECC9M+NAx+vqn9XpPRVbaoVz9ThoRS3bPfmyJhV5nlfUIEY9zy1ZLITqTc+8RTS/AwAA//+ODPq5DAQAAA=="
time=2026-03-04T15:46:11.063Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:46:11.063Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T15:47:11.070Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-04T15:47:11.562Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj2610048.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj2610048.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d databaseName:arohcpprow-rp-j2610048 databaseUrl:https://arohcpprow-rp-j2610048.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-1n42z7iz/pipeline@sha256:c0b52f4451b052ec0e26f17df6c414a1b8bd50fdf71ec8bed58bd55d2084ca5b replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-1n42z7iz/pipeline:pull workloadIdentityClientId:8b702cd7-2c4a-4e01-8dbe-6220f9c6da7f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T15:47:11.562Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:47:11.906Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-04T15:47:12.199Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:47:12.199Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:47:12.472Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy numVersions=1
time=2026-03-04T15:47:13.465Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:47:13.465Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=ServiceAccount" namespace=aro-hcp name=backend
time=2026-03-04T15:47:13.828Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=ConfigMap" namespace=aro-hcp name=backend-azure-runtime-config
time=2026-03-04T15:47:14.268Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=ConfigMap" namespace=aro-hcp name=backend-config
time=2026-03-04T15:47:14.389Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="rbac.authorization.k8s.io/v1, Kind=Role" namespace=aro-hcp name=leader-election
time=2026-03-04T15:47:14.837Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="rbac.authorization.k8s.io/v1, Kind=RoleBinding" namespace=aro-hcp name=backend-leader-election
time=2026-03-04T15:47:15.334Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=aro-hcp name=aro-hcp-backend-metrics
time=2026-03-04T15:47:15.456Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=aro-hcp name=aro-hcp-backend
time=2026-03-04T15:47:15.582Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="acrpull.microsoft.com/v1beta2, Kind=AcrPullBinding" namespace=aro-hcp name=backend-pull-binding
time=2026-03-04T15:47:15.834Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=aro-hcp name=allow-metrics-backend
time=2026-03-04T15:47:16.225Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="security.istio.io/v1beta1, Kind=PeerAuthentication" namespace=aro-hcp name=aro-hcp-backend-metrics
time=2026-03-04T15:47:16.542Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="secrets-store.csi.x-k8s.io/v1, Kind=SecretProviderClass" namespace=aro-hcp name=backend-service-key-vault
time=2026-03-04T15:47:16.896Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=aro-hcp name=aro-hcp-backend
time=2026-03-04T15:47:17.237Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-03-04T15:47:17.237Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:47:17.352Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy numVersions=1
time=2026-03-04T15:52:20.165Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T15:52:20.165Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:52:20.165Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:52:24.029Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Upgrade \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj2610048.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj2610048.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d databaseName:arohcpprow-rp-j2610048 databaseUrl:https://arohcpprow-rp-j2610048.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-1n42z7iz/pipeline@sha256:c0b52f4451b052ec0e26f17df6c414a1b8bd50fdf71ec8bed58bd55d2084ca5b replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-1n42z7iz/pipeline:pull workloadIdentityClientId:8b702cd7-2c4a-4e01-8dbe-6220f9c6da7f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T15:52:24.030Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5RTwW7aQBS8+yuefALJGxFKGokqB5r2VKVCbU+peljvTmHDep/1dg0i4uMrYyhWGlp6s2ZnxrPv7WQeiQSRGzGIdEdWJ5106THg8gkmfXLBTmMSFxYFddBnXeE3hDXCHom1Nkd4+D0jIsq/QtbOYGYMNyHlBeWlNisE235qYbU0dV503C/s0cIe2kIUPExyHM4w37tgXVj0LNW/hR9Qe95W6KIcDtX5SLMmLVncs2795uyd2e5Z3vNGVUjiTPyLfA5Ia4GQnNEvMh11R59X9IfpPXBwieWyyPccfrrFg677k9HPjUBJE5KroMyecqn2LPuQ7v+uNDMyb7x/ZXt1470qT/gffzOCNBdeOwu59zrGvjp2YdQKW7XWje8vOM9+vMtWTQkJSIgf2wcbsx1tlhBQO5KYdFVTibQBAg2sTmjhQT4ejd+q0Rs1mny7vplObqfXt4/5kK6u6BznZjwdTx7z4TDb0RO7QCsX7J0LAdKrGQc6tatfq5d9ynZUC7enp6AF1Ww78mUmBXleFFQhRr1oLVkshMpt7/IW0fwKAAD//6LgrtgMBAAA"
time=2026-03-04T15:52:24.032Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:52:24.032Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T15:53:24.034Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-04T15:53:24.611Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj2610048.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj2610048.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d databaseName:arohcpprow-rp-j2610048 databaseUrl:https://arohcpprow-rp-j2610048.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-1n42z7iz/pipeline@sha256:c0b52f4451b052ec0e26f17df6c414a1b8bd50fdf71ec8bed58bd55d2084ca5b replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-1n42z7iz/pipeline:pull workloadIdentityClientId:8b702cd7-2c4a-4e01-8dbe-6220f9c6da7f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T15:53:24.611Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:53:24.932Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-04T15:53:25.199Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:53:25.199Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:53:25.424Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy numVersions=2
time=2026-03-04T15:53:26.454Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:53:26.454Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=ServiceAccount" namespace=aro-hcp name=backend
time=2026-03-04T15:53:26.794Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=ConfigMap" namespace=aro-hcp name=backend-azure-runtime-config
time=2026-03-04T15:53:27.143Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=ConfigMap" namespace=aro-hcp name=backend-config
time=2026-03-04T15:53:27.320Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="rbac.authorization.k8s.io/v1, Kind=Role" namespace=aro-hcp name=leader-election
time=2026-03-04T15:53:27.774Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="rbac.authorization.k8s.io/v1, Kind=RoleBinding" namespace=aro-hcp name=backend-leader-election
time=2026-03-04T15:53:28.074Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="/v1, Kind=Service" namespace=aro-hcp name=aro-hcp-backend-metrics
time=2026-03-04T15:53:28.198Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="apps/v1, Kind=Deployment" namespace=aro-hcp name=aro-hcp-backend
time=2026-03-04T15:53:28.341Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="acrpull.microsoft.com/v1beta2, Kind=AcrPullBinding" namespace=aro-hcp name=backend-pull-binding
time=2026-03-04T15:53:28.679Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="security.istio.io/v1, Kind=AuthorizationPolicy" namespace=aro-hcp name=allow-metrics-backend
time=2026-03-04T15:53:29.007Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="security.istio.io/v1beta1, Kind=PeerAuthentication" namespace=aro-hcp name=aro-hcp-backend-metrics
time=2026-03-04T15:53:29.386Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="secrets-store.csi.x-k8s.io/v1, Kind=SecretProviderClass" namespace=aro-hcp name=backend-service-key-vault
time=2026-03-04T15:53:29.733Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=aro-hcp name=aro-hcp-backend
time=2026-03-04T15:53:30.024Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-03-04T15:53:30.024Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:53:30.141Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy numVersions=2
time=2026-03-04T15:58:32.984Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T15:58:32.984Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:58:32.984Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:58:37.241Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Upgrade \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj2610048.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj2610048.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d databaseName:arohcpprow-rp-j2610048 databaseUrl:https://arohcpprow-rp-j2610048.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-1n42z7iz/pipeline@sha256:c0b52f4451b052ec0e26f17df6c414a1b8bd50fdf71ec8bed58bd55d2084ca5b replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-1n42z7iz/pipeline:pull workloadIdentityClientId:8b702cd7-2c4a-4e01-8dbe-6220f9c6da7f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:11ca5fcd-a6a4-4c6e-b5c2-8e0828528f7d workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-04T15:58:37.242Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTQW8aMRCF7/srRnsCaR3R0LQVVQ407alKhZqeUvXgtV/BwetZjW0QET++WihlhUDl+vw974zfvsIjkSByFoNI92R10knXHgOuX2DSVxfsJCZxYV7RXvqmG/yTsELYKbHV5iAPfxZEROV39igrKj20hSh4mOQ4dJIWVgvTltWefIKsnEHvRNXaLBGsapDEmXjG9Bmt502DkM74zvAzQKY5LRCSM/pkkiu+9wQjSDPhlbOQB6/jjjoY434HtcRGrXT26fKejxxcYrlu7L+eqTGc96teZh84/HbzR932B9OvWaAkh+QaKLNDrvVepLtsP7lgXZj3+f9HPTUyy96f8bbZe1Uf9VNjTgsW97pLbsbemc2O8p7Xh9TOPWPx62OxzDUkICF+6X7YWGxpvYCAuieJSTct1UhrINDA6oROHpS3o9t3ajRWo7c/3txN7saT8ei5HNLNDV1kPkzG75/L4bDY0gu7QEsX7L0LAdKrGQc6tqtfq9M+FVtqhbvT46AVtWz38HWXVOR5XlGDGPW8u5LFQqje9Ja3iOZPAAAA//9xl/54DAQAAA=="
time=2026-03-04T15:58:37.246Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-04T15:58:37.246Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded"
time=2026-03-04T15:59:37.252Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7ec0c2c
 n/a provision failed #4255 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
14:47:01 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T15:19:40.232Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T15:19:42.448Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T15:19:43.595Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9c9ff51ae6dd810e90c78e645c16a1e8f40cf267875768850d4424dac4a39dea
time=2026-03-04T15:21:44.461Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6065536/providers/Microsoft.Resources/deployments/9c9ff51ae6dd810e90c78e645c16a1e8f40cf267875768850d4424dac4a39dea/operationStatuses/08584289693025784838\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6065536/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6065536/providers/Microsoft.Monitor/accounts/services-j6065536/providers/Microsoft.Authorization/roleAssignments/c5a107d0-e083-5808-9a23-2e89a91f723e\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T151957Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6772a42f-418d-4914-b2b3-caa66795dd17', request correlation id '5825ae10-b7aa-4247-8373-e848ee48a8d9'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head e6e11ea · merge 35511ce
 n/a unknown failed #4314 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b60ce1f · merge 138964d
 n/a provision failed #4226 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
14:20:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T14:49:52.535Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T14:49:54.872Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T14:49:56.801Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=84f1fcf9b1b78d09b21ea787e1156344193598fddfa40eb59946d0f2e4d17c84
time=2026-03-04T14:51:57.853Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2968064/providers/Microsoft.Resources/deployments/84f1fcf9b1b78d09b21ea787e1156344193598fddfa40eb59946d0f2e4d17c84/operationStatuses/08584289710899079350\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2968064/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2968064/providers/Microsoft.Monitor/accounts/hcps-j2968064/providers/Microsoft.Authorization/roleAssignments/13ebece5-8654-5bfa-b79f-8789153eeb50\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T145021Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '509927fe-068a-42fa-9c8d-d4429109c379', request correlation id 'a5d00aa0-eccf-4134-a243-97d535dd85c3'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b8a2fa0 · merge b8a2fa0
 n/a e2e failed #4309 (merged) 1
e2e (1)
Test Panicked
14:19:36 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000c24c80?, {0x7fcb2c1bad68?, 0xc000a45a70?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:603 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000c72240, {0x7fcb2c1bad68, 0xc000a45a70}, {0x781823f, 0x14}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func2.2({0x7fcb2c1bad68, 0xc000a45a70})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/admin_api.go:60 +0xf1
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8221420?, 0xc000a45a70?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 71
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000c24c80?, {0x7fcb2c1bad68?, 0xc000a45a70?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:603 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000c72240, {0x7fcb2c1bad68, 0xc000a45a70}, {0x781823f, 0x14}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func2.2({0x7fcb2c1bad68, 0xc000a45a70})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/admin_api.go:60 +0xf1
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8221420?, 0xc000a45a70?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 71
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2cf3195 · merge a36f37a
 n/a n/a passed #4205 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
13:20:29 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T13:21:40.266Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T13:21:42.503Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T13:21:43.711Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b0ef20b1e9322a8c9fe7ed039ecf1eff448586a1d470f852a0b87085ae907dd3
time=2026-03-04T13:23:44.583Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1168384/providers/Microsoft.Resources/deployments/b0ef20b1e9322a8c9fe7ed039ecf1eff448586a1d470f852a0b87085ae907dd3/operationStatuses/08584289763824543640\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1168384/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1168384/providers/Microsoft.Monitor/accounts/hcps-j1168384/providers/Microsoft.Authorization/roleAssignments/bbd32259-8b0e-5394-a94e-abe3679a5071\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T132205Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd319f7d5-b87c-4f1a-91f4-62d0f4db254e', request correlation id '7d61af07-ab05-42c3-83aa-6cde1db6d789'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 59174f2 · merge 87ae44a
 n/a e2e failed ⚠#4154 (merged) 1
e2e (1)
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Forbidden: field is immutable; provider Microsoft.RedHatOpenShift
12:33:11 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:210]: Unexpected error:
    <*errors.joinError | 0xc000a5a630>: 
    invalid VM size test error: PUT https://rp.j2077824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-pmfgpl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Forbidden: field is immutable",
            "target": "properties.platform"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Forbidden: field is immutable",
            "target": "properties.platform.vmSize"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0xc0015986c0>{
                msg: "invalid VM size test error: PUT https://rp.j2077824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-pmfgpl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"2 errors occurred\",\n    \"details\": [\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.vmSize\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc001589e30>{
                    ErrorCode: "MultipleErrorsOccurred",
                    StatusCode: 400,
                    RawResponse: {
                        Status: "400 Bad Request",
                        StatusCode: 400,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "Server": ["istio-envoy"],
                            "Content-Type": ["application/json"],
                            "X-Ms-Error-Code": [
                                "MultipleErrorsOccurred",
                            ],
                            "X-Ms-Request-Id": [
                                "d3e1ae62-af6f-4f56-9ed6-d0e37c1e8237",
                            ],
                            "Date": [
                                "Wed, 04 Mar 2026 13:53:38 GMT",
                            ],
                            "Content-Length": ["515"],
                            "X-Envoy-Upstream-Service-Time": ["131"],
                        },
                        Body: <*exported.nopClosingBytesReader | 0xc0015985e0>{
                            s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 112, 108, 97, 116, 102, 111, 114, 109, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:210]: Unexpected error:
    <*errors.joinError | 0xc000a5a630>: 
    invalid VM size test error: PUT https://rp.j2077824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-pmfgpl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Forbidden: field is immutable",
            "target": "properties.platform"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Forbidden: field is immutable",
            "target": "properties.platform.vmSize"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0xc0015986c0>{
                msg: "invalid VM size test error: PUT https://rp.j2077824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-pmfgpl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"2 errors occurred\",\n    \"details\": [\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.vmSize\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc001589e30>{
                    ErrorCode: "MultipleErrorsOccurred",
                    StatusCode: 400,
                    RawResponse: {
                        Status: "400 Bad Request",
                        StatusCode: 400,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "Server": ["istio-envoy"],
                            "Content-Type": ["application/json"],
                            "X-Ms-Error-Code": [
                                "MultipleErrorsOccurred",
                            ],
                            "X-Ms-Request-Id": [
                                "d3e1ae62-af6f-4f56-9ed6-d0e37c1e8237",
                            ],
                            "Date": [
                                "Wed, 04 Mar 2026 13:53:38 GMT",
                            ],
                            "Content-Length": ["515"],
                            "X-Envoy-Upstream-Service-Time": ["131"],
                        },
                        Body: <*exported.nopClosingBytesReader | 0xc0015985e0>{
                            s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 112, 108, 97, 116, 102, 111, 114, 109, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
12:32:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T12:59:17.588Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T12:59:19.482Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T12:59:20.587Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=cb14cf7492012f6e7f29e58cb3f6b6c850e54d3a78f7e0020964ae4d4544f68b
time=2026-03-04T13:01:21.445Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7896448/providers/Microsoft.Resources/deployments/cb14cf7492012f6e7f29e58cb3f6b6c850e54d3a78f7e0020964ae4d4544f68b/operationStatuses/08584289777255371328\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7896448/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7896448/providers/Microsoft.Monitor/accounts/services-j7896448/providers/Microsoft.Authorization/roleAssignments/48c66f00-12ec-5fee-8c21-59862a32ffbd\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T125943Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95038bf-e92c-4f31-bb62-6068506f3876', request correlation id '267d2c1b-8f93-4a0a-a28d-1cb57d6ae07c'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7896448/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7896448/providers/Microsoft.Monitor/accounts/hcps-j7896448/providers/Microsoft.Authorization/roleAssignments/10b99d6e-338a-5355-afce-a92eca48ebbd\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T125942Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'c4fa6d90-9d8c-4341-8c06-a2a9bb96ca2a', request correlation id '267d2c1b-8f93-4a0a-a28d-1cb57d6ae07c'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7ec0c2c
 n/a n/a passed #4255 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7148a34 · merge 7148a34
 n/a provision failed #4303 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
12:28:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T12:58:02.547Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T12:58:04.497Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T12:58:05.699Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b7608decf4d7e86e43b8ad772dc997e58019e2b2d7f6d61fad459af928bcb03d
time=2026-03-04T13:00:06.433Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5401472/providers/Microsoft.Resources/deployments/b7608decf4d7e86e43b8ad772dc997e58019e2b2d7f6d61fad459af928bcb03d/operationStatuses/08584289778004909795\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5401472/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5401472/providers/Microsoft.Monitor/accounts/hcps-j5401472/providers/Microsoft.Authorization/roleAssignments/a271e06e-5b1b-5d06-91ca-2b93e55309e3\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T125826Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7f7ac6f5-3c2d-4c73-b3c7-e32dba6f02a7', request correlation id 'c780276f-29b9-416a-afcc-859b0b146c24'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b8a2fa0 · merge b8a2fa0
 n/a n/a passed #4309 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 3900846
 n/a n/a passed #3696 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7ec0c2c
 n/a provision failed #4255 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
11:18:37 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T11:53:00.963Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T11:53:03.044Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T11:53:04.184Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=32bf687ec575978e9034d135870fdf361b601dc0ea10f169720c05d939c188df
time=2026-03-04T11:55:05.227Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7164160/providers/Microsoft.Resources/deployments/32bf687ec575978e9034d135870fdf361b601dc0ea10f169720c05d939c188df/operationStatuses/08584289817019116868\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7164160/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7164160/providers/Microsoft.Monitor/accounts/services-j7164160/providers/Microsoft.Authorization/roleAssignments/4bc99e86-4cd0-5209-82ab-b97d7371f03e\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T115349Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ca62049-817f-469b-8e30-9b5170f94808', request correlation id '287a5bea-bfb7-4d55-99c4-06bb805450d2'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7ec0c2c
 n/a provision failed #4255 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
11:15:39 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T11:49:19.351Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T11:49:22.079Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T11:49:23.388Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d60f8e60b9a20bcae04f5f163622acd095ae978be178d068a7b8719fa6586c2a
time=2026-03-04T11:51:24.456Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7907072/providers/Microsoft.Resources/deployments/d60f8e60b9a20bcae04f5f163622acd095ae978be178d068a7b8719fa6586c2a/operationStatuses/08584289819228353110\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7907072/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7907072/providers/Microsoft.Monitor/accounts/hcps-j7907072/providers/Microsoft.Authorization/roleAssignments/fc97917e-239a-519a-9fa3-5a6ae6fd8f6c\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T114939Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '4c802878-9ab7-4ca4-9e5e-ac80a0dcadd5', request correlation id '33bf18dd-afdd-4352-a41e-935583b05a05'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7907072/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7907072/providers/Microsoft.Monitor/accounts/services-j7907072/providers/Microsoft.Authorization/roleAssignments/d96906f2-df7b-519a-8c8f-7d3a4044ae9e\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T114945Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f10484e9-9811-4433-8f73-509048286ff8', request correlation id '33bf18dd-afdd-4352-a41e-935583b05a05'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 59174f2 · merge 87ae44a
 n/a provision failed #4154 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
11:11:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T11:49:23.487Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T11:49:25.491Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T11:49:26.819Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f1fd91dcc9b64f164c15a549c41f4610b624a6cac707401d6b57805d5c8165f4
time=2026-03-04T11:51:27.880Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5384704/providers/Microsoft.Resources/deployments/f1fd91dcc9b64f164c15a549c41f4610b624a6cac707401d6b57805d5c8165f4/operationStatuses/08584289819194411977\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5384704/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5384704/providers/Microsoft.Monitor/accounts/hcps-j5384704/providers/Microsoft.Authorization/roleAssignments/5d4955cf-43b7-562d-a752-eb6993861d8f\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T114941Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '307464a7-269c-462b-bf87-b3ce6fdd6e49', request correlation id '3203dd2c-29bf-4e66-a0c9-7607277718c8'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
11:06:09 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T11:35:27.488Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T11:35:30.012Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T11:35:31.421Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b540db4c2c415358f1ae9b1ef6f5fedf81a6c8a32a2f2b2563db748eb861cd19
time=2026-03-04T11:37:02.531Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1357440/providers/Microsoft.Resources/deployments/b540db4c2c415358f1ae9b1ef6f5fedf81a6c8a32a2f2b2563db748eb861cd19/operationStatuses/08584289827547348911\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1357440/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1357440/providers/Microsoft.Monitor/accounts/services-j1357440/providers/Microsoft.Authorization/roleAssignments/39d620df-9b16-51d5-a208-70f2672b162c\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T113554Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '53d127b6-a5cd-4234-8664-a74d870ec7c0', request correlation id '42097278-847f-4b3e-99ab-626a4e16016b'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a n/a passed #4243 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
11:00:01 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T11:37:48.994Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T11:37:52.405Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T11:37:53.534Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=76406f78b697f196105df3a716a0c7bff6e645ad76711fdcdecedba00afe6d73
time=2026-03-04T11:39:54.583Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9619584/providers/Microsoft.Resources/deployments/76406f78b697f196105df3a716a0c7bff6e645ad76711fdcdecedba00afe6d73/operationStatuses/08584289826126209316\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9619584/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9619584/providers/Microsoft.Monitor/accounts/hcps-j9619584/providers/Microsoft.Authorization/roleAssignments/08548882-11d3-5865-8fa8-9560d5772499\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T113817Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '68c6e2ab-212b-472f-b26e-de99e69a4d4c', request correlation id '5a8b2373-15ae-480c-9e15-0574d4c90dc0'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 59174f2 · merge 87ae44a
 n/a provision failed #4154 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
10:55:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T11:01:00.465Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T11:01:03.158Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T11:01:04.358Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=ea3a1d64c5c8a1a4baa898bd8e799908a9f8ad60c74d7f71957550080033c73b
time=2026-03-04T11:03:06.143Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9272960/providers/Microsoft.Resources/deployments/ea3a1d64c5c8a1a4baa898bd8e799908a9f8ad60c74d7f71957550080033c73b/operationStatuses/08584289848218121125\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9272960/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9272960/providers/Microsoft.Monitor/accounts/services-j9272960/providers/Microsoft.Authorization/roleAssignments/e67c5551-228f-545b-9aa8-193e6153e2b8\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T110121Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '30765463-f0b2-4f74-b998-e117d0e67a36', request correlation id '9328c495-d1b1-48e7-9af6-f2abf1bf0d71'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2441e9f · merge 40bc11d
 n/a e2e failed #4313 (merged) 18
e2e (18)
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f31a70>: 
    failed to create HCP cluster list-test-cluster-7wr5z5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: failed waiting for cluster="list-test-cluster-7wr5z5" in resourcegroup="cluster-listing-85q47f" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster list-test-cluster-7wr5z5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: failed waiting for cluster=\"list-test-cluster-7wr5z5\" in resourcegroup=\"cluster-listing-85q47f\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001076110>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f",
            },
            <*fmt.wrapErrors | 0xc000f31a40>{
                msg: "failed waiting for cluster=\"list-test-cluster-7wr5z5\" in resourcegroup=\"cluster-listing-85q47f\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001076110>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f31a70>: 
    failed to create HCP cluster list-test-cluster-7wr5z5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: failed waiting for cluster="list-test-cluster-7wr5z5" in resourcegroup="cluster-listing-85q47f" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster list-test-cluster-7wr5z5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: failed waiting for cluster=\"list-test-cluster-7wr5z5\" in resourcegroup=\"cluster-listing-85q47f\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001076110>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f",
            },
            <*fmt.wrapErrors | 0xc000f31a40>{
                msg: "failed waiting for cluster=\"list-test-cluster-7wr5z5\" in resourcegroup=\"cluster-listing-85q47f\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001076110>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-7wr5z5 in resource group cluster-listing-85q47f",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010395c0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-n9pws6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-n9pws6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00100c970>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6",
            },
            <*fmt.wrapErrors | 0xc001039590>{
                msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-n9pws6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00100c970>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010395c0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-n9pws6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-n9pws6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00100c970>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6",
            },
            <*fmt.wrapErrors | 0xc001039590>{
                msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-n9pws6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00100c970>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-n9pws6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc00149b290>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-94fb29" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-94fb29\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ce09a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29",
            },
            <*fmt.wrapErrors | 0xc00149b260>{
                msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-94fb29\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ce09a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc00149b290>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-94fb29" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-94fb29\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ce09a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29",
            },
            <*fmt.wrapErrors | 0xc00149b260>{
                msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-94fb29\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ce09a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-94fb29",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00178b0b0>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-8grqgl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8grqgl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ec38b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl",
            },
            <*fmt.wrapErrors | 0xc000ef8ea0>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8grqgl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ec38b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00178b0b0>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-8grqgl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8grqgl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ec38b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl",
            },
            <*fmt.wrapErrors | 0xc000ef8ea0>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8grqgl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ec38b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-8grqgl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012cd380>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-kks6bc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-kks6bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00131f040>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc",
            },
            <*fmt.wrapErrors | 0xc0012cd350>{
                msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-kks6bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00131f040>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012cd380>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-kks6bc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-kks6bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00131f040>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc",
            },
            <*fmt.wrapErrors | 0xc0012cd350>{
                msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-kks6bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00131f040>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-kks6bc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc001356660>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-w7zqgp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-w7zqgp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00132d850>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp",
            },
            <*fmt.wrapErrors | 0xc0013565d0>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-w7zqgp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00132d850>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc001356660>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-w7zqgp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-w7zqgp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00132d850>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp",
            },
            <*fmt.wrapErrors | 0xc0013565d0>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-w7zqgp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00132d850>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-w7zqgp",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00038aed0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-blmgc7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-blmgc7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001065e40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7",
            },
            <*fmt.wrapErrors | 0xc00038aea0>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-blmgc7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001065e40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00038aed0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-blmgc7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-blmgc7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001065e40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7",
            },
            <*fmt.wrapErrors | 0xc00038aea0>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-blmgc7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001065e40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-blmgc7",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc00159e8d0>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-cjg6hw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cjg6hw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000efae40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw",
            },
            <*fmt.wrapErrors | 0xc00159e8a0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cjg6hw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000efae40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc00159e8d0>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-cjg6hw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cjg6hw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000efae40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw",
            },
            <*fmt.wrapErrors | 0xc00159e8a0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cjg6hw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000efae40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-cjg6hw",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Deserializaion Error: no output from command
09:58:34 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
> github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7fbe2ee580b0, 0xc0008f0ea0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8221420?, 0xc0008f0ea0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 85
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/04/26 12:20:17.181
  "ts"="2026-03-04 12:20:17.181376" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-04 12:20:17.181449" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-r8299g" "resourceGroup"="customer-rg-fmprdk"
  "ts"="2026-03-04 12:20:17.754133" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-r8299g" "resourceGroup"=""
  "ts"="2026-03-04 12:20:18.618579" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-gt5nj2" "resourceGroup"="customer-rg-fmprdk"
  "ts"="2026-03-04 12:20:18.844790" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-gt5nj2" "resourceGroup"=""
  "ts"="2026-03-04 12:20:19.791630" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 12:20:19.791746" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 12:20:19.791815" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-04 12:20:19.792915" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-108"
  "ts"="2026-03-04 12:20:19.792942" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-124"
  "ts"="2026-03-04 12:20:19.792927" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-136"
  "ts"="2026-03-04 12:20:19.792966" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-fmprdk"
  "ts"="2026-03-04 12:20:20.465045" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-108" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 12:20:20.752891" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-124" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 12:20:21.092977" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-136" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 12:20:21.529002" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-04 12:20:21.896103" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-108" "deployment"="msi-deployment-108"
  "ts"="2026-03-04 12:20:22.262610" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-124" "deployment"="msi-deployment-124"
  "ts"="2026-03-04 12:20:22.786160" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-136" "deployment"="msi-deployment-136"
  "ts"="2026-03-04 12:20:22.976200" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="customer-infra-basic-hcp-cluster2-gt5nj2"
  "ts"="2026-03-04 12:20:24.393749" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-04 12:20:24.855086" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="customer-infra-basic-hcp-cluster-r8299g"
  "ts"="2026-03-04 12:20:25.388961" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-04 12:20:25.389089" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-04 12:20:25.389193" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-fmprdk"
Command Error: signal: killed
Deserializaion Error: no output from command
> github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7fbe2ee580b0, 0xc0008f0ea0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8221420?, 0xc0008f0ea0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 85
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/04/26 12:20:17.181
  "ts"="2026-03-04 12:20:17.181376" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-04 12:20:17.181449" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-r8299g" "resourceGroup"="customer-rg-fmprdk"
  "ts"="2026-03-04 12:20:17.754133" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-r8299g" "resourceGroup"=""
  "ts"="2026-03-04 12:20:18.618579" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-gt5nj2" "resourceGroup"="customer-rg-fmprdk"
  "ts"="2026-03-04 12:20:18.844790" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-gt5nj2" "resourceGroup"=""
  "ts"="2026-03-04 12:20:19.791630" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 12:20:19.791746" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 12:20:19.791815" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-04 12:20:19.792915" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-108"
  "ts"="2026-03-04 12:20:19.792942" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-124"
  "ts"="2026-03-04 12:20:19.792927" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-136"
  "ts"="2026-03-04 12:20:19.792966" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-fmprdk"
  "ts"="2026-03-04 12:20:20.465045" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-108" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 12:20:20.752891" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-124" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 12:20:21.092977" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-136" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 12:20:21.529002" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-04 12:20:21.896103" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-108" "deployment"="msi-deployment-108"
  "ts"="2026-03-04 12:20:22.262610" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-124" "deployment"="msi-deployment-124"
  "ts"="2026-03-04 12:20:22.786160" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-136" "deployment"="msi-deployment-136"
  "ts"="2026-03-04 12:20:22.976200" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="customer-infra-basic-hcp-cluster2-gt5nj2"
  "ts"="2026-03-04 12:20:24.393749" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-04 12:20:24.855086" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-fmprdk" "deployment"="customer-infra-basic-hcp-cluster-r8299g"
  "ts"="2026-03-04 12:20:25.388961" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-04 12:20:25.389089" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-04 12:20:25.389193" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-fmprdk"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a29950>: 
    failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-gdsz9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-gdsz9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001136530>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k",
            },
            <*fmt.wrapErrors | 0xc000a29920>{
                msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-gdsz9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001136530>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a29950>: 
    failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-gdsz9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-gdsz9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001136530>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k",
            },
            <*fmt.wrapErrors | 0xc000a29920>{
                msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-gdsz9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001136530>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-gdsz9k",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc001585c80>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-t2t7rc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-t2t7rc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a7f300>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc",
            },
            <*fmt.wrapErrors | 0xc001585c50>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-t2t7rc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000a7f300>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc001585c80>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-t2t7rc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-t2t7rc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a7f300>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc",
            },
            <*fmt.wrapErrors | 0xc001585c50>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-t2t7rc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000a7f300>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-t2t7rc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc001289290>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-vftvd8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vftvd8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010bc570>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8",
            },
            <*fmt.wrapErrors | 0xc001289260>{
                msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vftvd8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010bc570>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc001289290>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-vftvd8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vftvd8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010bc570>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8",
            },
            <*fmt.wrapErrors | 0xc001289260>{
                msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vftvd8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010bc570>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-vftvd8",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:100]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010d0f00>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-ww94ws" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-ww94ws\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011bc520>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws",
            },
            <*fmt.wrapErrors | 0xc0010d0ed0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-ww94ws\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011bc520>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:100]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010d0f00>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-ww94ws" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-ww94ws\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011bc520>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws",
            },
            <*fmt.wrapErrors | 0xc0010d0ed0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-ww94ws\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011bc520>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ww94ws",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010f49f0>: 
    failed to create HCP cluster ea-listsf9cdz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: failed waiting for cluster="ea-listsf9cdz" in resourcegroup="ea-list-8r6rmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listsf9cdz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: failed waiting for cluster=\"ea-listsf9cdz\" in resourcegroup=\"ea-list-8r6rmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fc9c10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq",
            },
            <*fmt.wrapErrors | 0xc0010f49c0>{
                msg: "failed waiting for cluster=\"ea-listsf9cdz\" in resourcegroup=\"ea-list-8r6rmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fc9c10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010f49f0>: 
    failed to create HCP cluster ea-listsf9cdz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: failed waiting for cluster="ea-listsf9cdz" in resourcegroup="ea-list-8r6rmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listsf9cdz, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: failed waiting for cluster=\"ea-listsf9cdz\" in resourcegroup=\"ea-list-8r6rmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fc9c10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq",
            },
            <*fmt.wrapErrors | 0xc0010f49c0>{
                msg: "failed waiting for cluster=\"ea-listsf9cdz\" in resourcegroup=\"ea-list-8r6rmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fc9c10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listsf9cdz in resource group ea-list-8r6rmq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Cluster should become ready within 45 minutes
09:58:34 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapErrors | 0xc001480c00>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lp4tkb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lp4tkb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f49a60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb",
            },
            <*fmt.wrapErrors | 0xc001480bd0>{
                msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lp4tkb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f49a60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapErrors | 0xc001480c00>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lp4tkb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lp4tkb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f49a60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb",
            },
            <*fmt.wrapErrors | 0xc001480bd0>{
                msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lp4tkb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f49a60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-lp4tkb",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a656e0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nnns7t" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nnns7t\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00110f8a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t",
            },
            <*fmt.wrapErrors | 0xc000a656b0>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nnns7t\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00110f8a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a656e0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nnns7t" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nnns7t\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00110f8a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t",
            },
            <*fmt.wrapErrors | 0xc000a656b0>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nnns7t\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00110f8a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-nnns7t",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:58:34 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc00151f380>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-ffhbdr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ffhbdr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e7a020>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr",
            },
            <*fmt.wrapErrors | 0xc00151f350>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ffhbdr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e7a020>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc00151f380>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-ffhbdr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ffhbdr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e7a020>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr",
            },
            <*fmt.wrapErrors | 0xc00151f350>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ffhbdr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e7a020>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-ffhbdr",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4b5ae23 · merge 4b5ae23
 n/a provision failed ★#4310 (merged) 1
provision (1)
post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress context deadline exceeded
09:43:29 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph
Show raw failure
time=2026-03-04T10:30:13.969Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-03-04T10:30:14.687Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-04T10:30:14.687Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:30:14.748Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine
time=2026-03-04T10:30:14.867Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false
time=2026-03-04T10:30:14.867Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:30:14.981Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:35:41.197Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T10:35:41.197Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:35:41.198Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:35:45.724Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Release \"mce-config\" failed: failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-04T10:35:45.726Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yVT2/aTBDG7/4UI59AsiNCyHvwqxxo2lPUVmp6StXDencwG9Yz1u4CQuLDV4tt7EIoBOXA5WGe386fXU9k0INFx0sr0cEDKOGFF7nBAeevKP2TJpU5bzUVCdTSN1HiXsIV0k5xlZCtPPwVAQDEj2bpPNofbDBOIOYKKZW1lpaCRIElks8WtWTQp0IpplQyecvGoE1X4+CMk3cBCyuzRrZN6BWEio2Wmyajs7RPmpSm4sPrvIB7UG7eOa7nnSj+DXbbysLKNPxqZ1Nh27Byabxuj0IqNOEp/wWWz1gZ3oRcQ+D5tl6MOsq/Q13LqSxXohCezxGmSn2nDvPINNO7RteZzNl5VCGhRv4Hqhn31/1Id+xgqu1tbn9Xd95d8AotCZLYEmZWlLhmu+gDntGutMSplLz8gBEd447G5MRVjLO2gwdz8or3nsX7aZcDnvZtnIajuytiWArTvtM3hEN75ywM58L0p1fPXTW34Bl33ZpvKrRurme+96Fo+W5nj37/Hy2WOVpCj+5L2Asu2sJ6jhbB6xKdF2UFOfo1IsFACY9BHsTj0fi/dHSXjiY/b0fZ3Si7nbzEQ7i5gZMx99nk/iUeDqMtvLImWGhSD5oIbW+bMUG3xPrb63BtRVuoLId/u0QTqFjVwZdBEjBcJFCic6IISLYKLeSbXvEKnfwTAAD///fnqy1zBwAA"
time=2026-03-04T10:35:45.727Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:35:45.727Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config step=deploy-mce-config error="error running Helm release deployment Step, failed to deploy helm release: failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T10:36:45.732Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-03-04T10:36:51.428Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-04T10:36:51.428Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:36:51.929Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine
time=2026-03-04T10:36:52.104Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:36:52.104Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:36:52.275Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config numVersions=1
time=2026-03-04T10:36:53.274Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:36:53.274Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="/v1, Kind=ServiceAccount" namespace=multicluster-engine name=klusterlet-addon-controller-v2
time=2026-03-04T10:36:53.594Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="/v1, Kind=ServiceAccount" namespace=multicluster-engine name=grc-policy-addon-sa
time=2026-03-04T10:36:53.760Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="/v1, Kind=ServiceAccount" namespace=multicluster-engine name=grc-sa
time=2026-03-04T10:36:53.928Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=open-cluster-management:klusterlet-addon-controller-v2
time=2026-03-04T10:36:54.414Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=open-cluster-management:grc:clusterrole
time=2026-03-04T10:36:54.537Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=open-cluster-management:grc:policyaddon-clusterrole
time=2026-03-04T10:36:54.659Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=open-cluster-management:klusterlet-addon-controller-v2
time=2026-03-04T10:36:54.966Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace=multicluster-engine name=open-cluster-management:grc:clusterrolebinding
time=2026-03-04T10:36:55.206Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=open-cluster-management:grc:policyaddon-clusterrolebinding
time=2026-03-04T10:36:55.324Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=Role" namespace=multicluster-engine name=grc-grc-policy-addon-role
time=2026-03-04T10:36:55.771Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=Role" namespace=multicluster-engine name=grc-grc-role
time=2026-03-04T10:36:55.889Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=RoleBinding" namespace=multicluster-engine name=grc-grc-policy-addon-rolebinding
time=2026-03-04T10:36:56.256Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=RoleBinding" namespace=multicluster-engine name=grc-grc-rolebinding
time=2026-03-04T10:36:56.380Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="apps/v1, Kind=Deployment" namespace=multicluster-engine name=klusterlet-addon-controller-v2
time=2026-03-04T10:36:56.477Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=96.223278ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/klusterlet-addon-controller-v2
time=2026-03-04T10:36:56.677Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=141.295013ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/klusterlet-addon-controller-v2
time=2026-03-04T10:36:56.975Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="apps/v1, Kind=Deployment" namespace=multicluster-engine name=grc-policy-addon-controller
time=2026-03-04T10:36:57.141Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="apps/v1, Kind=Deployment" namespace=multicluster-engine name=grc-policy-propagator
time=2026-03-04T10:36:57.277Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=135.890681ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/grc-policy-propagator
time=2026-03-04T10:36:57.477Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=141.283962ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/grc-policy-propagator
time=2026-03-04T10:36:57.543Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="addon.open-cluster-management.io/v1alpha1, Kind=AddOnDeploymentConfig" namespace=multicluster-engine name=addon-hosted-config
time=2026-03-04T10:36:57.677Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=133.297092ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/namespaces/multicluster-engine/addondeploymentconfigs/addon-hosted-config
time=2026-03-04T10:36:57.877Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=141.285455ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/namespaces/multicluster-engine/addondeploymentconfigs/addon-hosted-config
time=2026-03-04T10:36:58.110Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="addon.open-cluster-management.io/v1alpha1, Kind=ClusterManagementAddOn" namespace="" name=config-policy-controller
time=2026-03-04T10:36:58.277Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=108.283574ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/clustermanagementaddons/config-policy-controller
time=2026-03-04T10:36:58.512Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="addon.open-cluster-management.io/v1alpha1, Kind=ClusterManagementAddOn" namespace="" name=governance-policy-framework
time=2026-03-04T10:36:58.677Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=105.252017ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/clustermanagementaddons/governance-policy-framework
time=2026-03-04T10:36:58.740Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="agent.open-cluster-management.io/v1, Kind=KlusterletAddonConfig" namespace=local-cluster name=local-cluster
time=2026-03-04T10:36:58.877Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=137.303197ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/agent.open-cluster-management.io/v1/namespaces/local-cluster/klusterletaddonconfigs/local-cluster
time=2026-03-04T10:36:59.077Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=141.281084ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/agent.open-cluster-management.io/v1/namespaces/local-cluster/klusterletaddonconfigs/local-cluster
time=2026-03-04T10:36:59.300Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="config.open-cluster-management.io/v1alpha1, Kind=KlusterletConfig" namespace="" name=global
time=2026-03-04T10:36:59.477Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=118.265276ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/config.open-cluster-management.io/v1alpha1/klusterletconfigs/global
time=2026-03-04T10:36:59.692Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="cluster.open-cluster-management.io/v1beta2, Kind=ManagedClusterSet" namespace="" name=hypershift-management-clusters
time=2026-03-04T10:36:59.877Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=125.252738ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/cluster.open-cluster-management.io/v1beta2/managedclustersets/hypershift-management-clusters
time=2026-03-04T10:37:00.134Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false
time=2026-03-04T10:37:00.134Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:37:00.250Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config numVersions=1
time=2026-03-04T10:42:07.958Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T10:42:07.958Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:42:07.958Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:42:12.269Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Upgrade \"mce-config\" failed: post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-04T10:42:12.271Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yVwW/aMBTG7/krrJxASqqUVpuUqQfW7VR1k9adOu3g2A/j4rwX2QaExB8/mSQkhbFQ1lv1+X2/fs/P5EUGPLPgaGkFOHbHJPfc88LAiIoXEP5Bo8ydtxpVwmrpGy9hL8EKcKe4iotWHv+KGGMsfjBL58Ea8PeEM63ihMXKUMFN+CtO6qonsCstYCoELdGHk8Xel3IpCVNB6C0ZAzZdTUJFuTRei7osBVQa4TROWZFWZLTYNDjHL2IM2n6Qgc8apUbVeo7+tyUDRVfzdtr5gEeOXIG8rw+fYNfIfFOBdXM982m5Oy8BfdoAXH8ujS/ECDJVgG1dz5oPD+uNQGVF3si2Kb2AUF96k2iQ1rvnd+3zDO5Bu73RXs470fxf2O1VnnyrZzzSw9c5YPkClaFNyPrfv/XXqKP8HepSTmWp4op7GiJMpfyOHab73tVJ5uQ8yBCokf+Basb9uB/pjh1Mtb3N9rq7YbeiFVjkKKAlzCwvYU120Qd0H+1piN51Ykhw0z6nYyH6/SlaLAuwCB7c17AXXLRl6zlYYF6X4DwvK1aAXwMgG0nuIcijeJJNPqTZTZrd/rzO8puPeZY9x2N2dcVO1dxO8uvJczweR1v2QhrZQqO804hge9uMkHVLrL+9DtdWtGWVpXDaBU1YRbIuPg+SMEMqYSU4x1VAkpVgWbHpNS/BiT8BAAD//3mvoTFzBwAA"
time=2026-03-04T10:42:12.273Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:42:12.274Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config step=deploy-mce-config error="error running Helm release deployment Step, failed to deploy helm release: post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T10:43:12.276Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-03-04T10:43:13.034Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-04T10:43:13.034Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:43:13.324Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine
time=2026-03-04T10:43:13.550Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:43:13.550Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:43:13.721Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config numVersions=2
time=2026-03-04T10:43:14.612Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:43:14.612Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="/v1, Kind=ServiceAccount" namespace=multicluster-engine name=klusterlet-addon-controller-v2
time=2026-03-04T10:43:14.893Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="/v1, Kind=ServiceAccount" namespace=multicluster-engine name=grc-policy-addon-sa
time=2026-03-04T10:43:15.059Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="/v1, Kind=ServiceAccount" namespace=multicluster-engine name=grc-sa
time=2026-03-04T10:43:15.227Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=open-cluster-management:klusterlet-addon-controller-v2
time=2026-03-04T10:43:15.724Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=open-cluster-management:grc:clusterrole
time=2026-03-04T10:43:15.851Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=open-cluster-management:grc:policyaddon-clusterrole
time=2026-03-04T10:43:15.974Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=open-cluster-management:klusterlet-addon-controller-v2
time=2026-03-04T10:43:16.275Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace=multicluster-engine name=open-cluster-management:grc:clusterrolebinding
time=2026-03-04T10:43:16.508Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=open-cluster-management:grc:policyaddon-clusterrolebinding
time=2026-03-04T10:43:16.626Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=Role" namespace=multicluster-engine name=grc-grc-policy-addon-role
time=2026-03-04T10:43:17.166Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=Role" namespace=multicluster-engine name=grc-grc-role
time=2026-03-04T10:43:17.289Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=RoleBinding" namespace=multicluster-engine name=grc-grc-policy-addon-rolebinding
time=2026-03-04T10:43:17.605Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="rbac.authorization.k8s.io/v1, Kind=RoleBinding" namespace=multicluster-engine name=grc-grc-rolebinding
time=2026-03-04T10:43:17.723Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="apps/v1, Kind=Deployment" namespace=multicluster-engine name=klusterlet-addon-controller-v2
time=2026-03-04T10:43:17.815Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=91.255291ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/klusterlet-addon-controller-v2
time=2026-03-04T10:43:18.015Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=139.301768ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/klusterlet-addon-controller-v2
time=2026-03-04T10:43:18.228Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="apps/v1, Kind=Deployment" namespace=multicluster-engine name=grc-policy-addon-controller
time=2026-03-04T10:43:18.414Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=125.284813ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/grc-policy-addon-controller
time=2026-03-04T10:43:18.478Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="apps/v1, Kind=Deployment" namespace=multicluster-engine name=grc-policy-propagator
time=2026-03-04T10:43:18.614Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=136.27702ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/grc-policy-propagator
time=2026-03-04T10:43:18.815Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=140.259156ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/apps/v1/namespaces/multicluster-engine/deployments/grc-policy-propagator
time=2026-03-04T10:43:18.878Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="addon.open-cluster-management.io/v1alpha1, Kind=AddOnDeploymentConfig" namespace=multicluster-engine name=addon-hosted-config
time=2026-03-04T10:43:19.015Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=136.303591ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/namespaces/multicluster-engine/addondeploymentconfigs/addon-hosted-config
time=2026-03-04T10:43:19.214Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=138.355599ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/namespaces/multicluster-engine/addondeploymentconfigs/addon-hosted-config
time=2026-03-04T10:43:19.526Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="addon.open-cluster-management.io/v1alpha1, Kind=ClusterManagementAddOn" namespace="" name=config-policy-controller
time=2026-03-04T10:43:19.849Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="addon.open-cluster-management.io/v1alpha1, Kind=ClusterManagementAddOn" namespace="" name=governance-policy-framework
time=2026-03-04T10:43:20.014Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=105.268597ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/addon.open-cluster-management.io/v1alpha1/clustermanagementaddons/governance-policy-framework
time=2026-03-04T10:43:20.075Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="agent.open-cluster-management.io/v1, Kind=KlusterletAddonConfig" namespace=local-cluster name=local-cluster
time=2026-03-04T10:43:20.215Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=139.295867ms reason="client-side throttling, not priority and fairness" verb=GET URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/agent.open-cluster-management.io/v1/namespaces/local-cluster/klusterletaddonconfigs/local-cluster
time=2026-03-04T10:43:20.414Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=138.236517ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/agent.open-cluster-management.io/v1/namespaces/local-cluster/klusterletaddonconfigs/local-cluster
time=2026-03-04T10:43:20.657Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="config.open-cluster-management.io/v1alpha1, Kind=KlusterletConfig" namespace="" name=global
time=2026-03-04T10:43:20.814Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=97.280774ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/config.open-cluster-management.io/v1alpha1/klusterletconfigs/global
time=2026-03-04T10:43:21.034Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config gvk="cluster.open-cluster-management.io/v1beta2, Kind=ManagedClusterSet" namespace="" name=hypershift-management-clusters
time=2026-03-04T10:43:21.214Z level=DEBUG+1 msg="Waited before sending request" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config delay=120.320229ms reason="client-side throttling, not priority and fairness" verb=PUT URL=https://prow-j5932672-mgmt-1-ykb7rkd7.hcp.westus3.azmk8s.io:443/apis/cluster.open-cluster-management.io/v1beta2/managedclustersets/hypershift-management-clusters
time=2026-03-04T10:43:21.510Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false
time=2026-03-04T10:43:21.510Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:43:21.677Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config numVersions=2
time=2026-03-04T10:48:29.332Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T10:48:29.332Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:48:29.332Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:48:34.292Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Upgrade \"mce-config\" failed: post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-04T10:48:34.294Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yVzW7bOhCF93qKgVYxIAWOE1xcqMgiTbsK2gJNVym6oMgJzZiaEUjahgE/fEFLslT/1K6RhTfHPJ/OcDiYxGIAh57nTqKHe1AiiCBKi1dcvqEMT4ZU4YMzpDNopK+iwq2EC6SN4mshO3n0MwEASB/t3Ad039limkHKNVIuGy2vBAmNFVIoZo1kMeRCKaZcMgXH1qLLF5PoTLN/Amoni1Z27dELCDVbI1dtosO0DqOdzOOvsbRVdIeruQ2m+waSNoTH/GdYPmFteRVDxoOnr+5s1F7+HnUpp3ZcCy0CnyI8KPWNeswj06vR0dMkmbIPqGKgVv4Lqm3wl20vN+xoauxdtj+rO+3WvEBHgiR2hFcnKlyymw0BT9uGPMTofSWWpbDdOzog7D3Pj4aUIf3ug3MGd2d+yt5xOe/INB1g70CPztbAemLEDtDOB/Qd7ZupLZfCDkM3L0a19/KMm1GYrmp0fmpew+A2uur90P6MbmEkPkjJ83eY7H3c3g16cRHjuC359SGZzUt0hAH957gXfLKG5RQdQjAV+iCqGkoMS0SCKyUCRvkqnYwn/+Xj23x89+NmXNzdFpObl3QE19dw9Mz/xe3dSzoaJWt4Y0MwM6TuDRG6wTZjgn6JDbfX7tpK1lA7jv/2QTOoWTWHz4NkYFlnUKH3QkckO4UOytWgeIVe/g4AAP//yqEAiXMHAAA="
time=2026-03-04T10:48:34.295Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-04T10:48:34.295Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config step=deploy-mce-config error="error running Helm release deployment Step, failed to deploy helm release: post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-04T10:49:34.303Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: post-upgrade hooks failed: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9026867
 n/a provision failed #4312 (closed) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
09:18:27 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T09:42:31.190Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T09:42:33.466Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T09:42:34.627Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b8cb79f8451e822112431e08375e086920b2c10cb6ed0a737b16df920f754435
time=2026-03-04T09:44:05.376Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5846912/providers/Microsoft.Resources/deployments/b8cb79f8451e822112431e08375e086920b2c10cb6ed0a737b16df920f754435/operationStatuses/08584289895315685810\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5846912/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5846912/providers/Microsoft.Monitor/accounts/hcps-j5846912/providers/Microsoft.Authorization/roleAssignments/ed28c9fa-e3ec-515e-83d1-fbc466c6a362\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T094247Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '50d2b453-bde4-4089-8d1b-88157a91f991', request correlation id 'f013c198-3840-4975-bd48-c454b9d90887'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7dc3871 · merge 7dc3871
 n/a n/a passed #4304 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7dc3871 · merge 7dc3871
 n/a n/a passed #4304 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 6420533 · merge 6420533
 n/a provision failed #4297 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
08:53:52 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T09:25:10.209Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T09:25:12.906Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T09:25:14.221Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=c9c13d8d3f177007ffa536c5a0c0178407f8c3dd5273a8d806544a5349af29e9
time=2026-03-04T09:26:45.099Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8196480/providers/Microsoft.Resources/deployments/c9c13d8d3f177007ffa536c5a0c0178407f8c3dd5273a8d806544a5349af29e9/operationStatuses/08584289905720284803\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8196480/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8196480/providers/Microsoft.Monitor/accounts/services-j8196480/providers/Microsoft.Authorization/roleAssignments/320965c8-f75a-575e-a9d7-0d05650a39e9\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T092535Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '060439e0-150f-427c-a38c-a333f76e592c', request correlation id '1236440e-4047-46d3-ab9d-a34c63bf4c6a'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8196480/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8196480/providers/Microsoft.Monitor/accounts/hcps-j8196480/providers/Microsoft.Authorization/roleAssignments/d9a62d63-ca67-5e4e-aac9-c9f7aaa090b9\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T092534Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd57cfc36-963c-4c2f-afc2-f0fb92926b36', request correlation id '1236440e-4047-46d3-ab9d-a34c63bf4c6a'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7ec0c2c
 n/a n/a passed #4255 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a n/a passed #4243 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6420533 · merge 6420533
 n/a n/a passed #4297 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 6420533 · merge 6420533
 n/a e2e failed #4297 (merged) 1
e2e (1)
Deserializaion Error: no output from command
05:53:48 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
oller.go:284
          > github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7f7c2de57260, 0xc00064ec00})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8221420?, 0xc00064ec00?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 89
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/04/26 08:18:21.788
  "ts"="2026-03-04 08:18:21.788612" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-04 08:18:21.788697" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-t9b7gc" "resourceGroup"="customer-rg-86vcln"
  "ts"="2026-03-04 08:18:22.518338" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-t9b7gc" "resourceGroup"=""
  "ts"="2026-03-04 08:18:23.236416" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-h765xv" "resourceGroup"="customer-rg-86vcln"
  "ts"="2026-03-04 08:18:23.392750" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-h765xv" "resourceGroup"=""
  "ts"="2026-03-04 08:18:24.093594" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 08:18:24.093734" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 08:18:24.093823" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-04 08:18:24.095674" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-72"
  "ts"="2026-03-04 08:18:24.095703" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-21"
  "ts"="2026-03-04 08:18:24.095721" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-86vcln"
  "ts"="2026-03-04 08:18:24.095694" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-56"
  "ts"="2026-03-04 08:18:24.744965" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-72" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 08:18:25.228603" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-04 08:18:25.539061" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-21" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 08:18:25.977987" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-56" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 08:18:26.430427" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-72" "deployment"="msi-deployment-72"
  "ts"="2026-03-04 08:18:26.742963" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="customer-infra-basic-hcp-cluster2-h765xv"
  "ts"="2026-03-04 08:18:27.218185" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-21" "deployment"="msi-deployment-21"
  "ts"="2026-03-04 08:18:27.640141" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-56" "deployment"="msi-deployment-56"
  "ts"="2026-03-04 08:18:28.418117" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-04 08:18:29.600189" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="customer-infra-basic-hcp-cluster-t9b7gc"
  "ts"="2026-03-04 08:18:30.278630" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-04 08:18:30.278790" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-04 08:18:30.278935" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-86vcln"
Command Error: signal: killed
Deserializaion Error: no output from command
oller.go:284
          > github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7f7c2de57260, 0xc00064ec00})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8221420?, 0xc00064ec00?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 89
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/04/26 08:18:21.788
  "ts"="2026-03-04 08:18:21.788612" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-04 08:18:21.788697" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-t9b7gc" "resourceGroup"="customer-rg-86vcln"
  "ts"="2026-03-04 08:18:22.518338" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-t9b7gc" "resourceGroup"=""
  "ts"="2026-03-04 08:18:23.236416" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-h765xv" "resourceGroup"="customer-rg-86vcln"
  "ts"="2026-03-04 08:18:23.392750" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-h765xv" "resourceGroup"=""
  "ts"="2026-03-04 08:18:24.093594" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 08:18:24.093734" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-04 08:18:24.093823" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-04 08:18:24.095674" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-72"
  "ts"="2026-03-04 08:18:24.095703" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-21"
  "ts"="2026-03-04 08:18:24.095721" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-86vcln"
  "ts"="2026-03-04 08:18:24.095694" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-56"
  "ts"="2026-03-04 08:18:24.744965" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-72" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 08:18:25.228603" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-04 08:18:25.539061" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-21" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 08:18:25.977987" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-56" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-04 08:18:26.430427" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-72" "deployment"="msi-deployment-72"
  "ts"="2026-03-04 08:18:26.742963" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="customer-infra-basic-hcp-cluster2-h765xv"
  "ts"="2026-03-04 08:18:27.218185" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-21" "deployment"="msi-deployment-21"
  "ts"="2026-03-04 08:18:27.640141" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-56" "deployment"="msi-deployment-56"
  "ts"="2026-03-04 08:18:28.418117" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-04 08:18:29.600189" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-86vcln" "deployment"="customer-infra-basic-hcp-cluster-t9b7gc"
  "ts"="2026-03-04 08:18:30.278630" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-04 08:18:30.278790" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-04 08:18:30.278935" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-86vcln"
Command Error: signal: killed
Deserializaion Error: no output from command
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7ec0c2c
 n/a provision failed #4255 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
05:04:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T05:36:43.059Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T05:36:45.713Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T05:36:46.792Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f4ed8eb32badc7234c2155c9ffa861442377110d421fd08fdfad2617b9a92a43
time=2026-03-04T05:38:47.732Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9482496/providers/Microsoft.Resources/deployments/f4ed8eb32badc7234c2155c9ffa861442377110d421fd08fdfad2617b9a92a43/operationStatuses/08584290042793385860\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9482496/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9482496/providers/Microsoft.Monitor/accounts/services-j9482496/providers/Microsoft.Authorization/roleAssignments/e7b2660f-1cf9-5e64-9dd3-68818bc1c9ac\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T053708Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a83d79bb-5c51-4dfa-939b-d0128782244e', request correlation id '5791217c-c4d5-4a2c-8730-912db8c89cd9'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7ec0c2c
 n/a provision failed #4255 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
05:01:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T05:36:41.406Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T05:36:44.342Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T05:36:45.509Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=163514b01592d3ff666266d7127ae3698b6d1f90d603e58fb1d301b0e6c30523
time=2026-03-04T05:38:16.486Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5669504/providers/Microsoft.Resources/deployments/163514b01592d3ff666266d7127ae3698b6d1f90d603e58fb1d301b0e6c30523/operationStatuses/08584290042806728132\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5669504/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5669504/providers/Microsoft.Monitor/accounts/hcps-j5669504/providers/Microsoft.Authorization/roleAssignments/58c9e49f-6d6b-5f1a-953e-551b7bd6d4af\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T053700Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b67f1b3a-6678-446b-a192-5358792debb0', request correlation id '4612beac-82e5-41b3-ad78-1e5a9ed71d19'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b8a2fa0 · merge b8a2fa0
 n/a n/a passed #4309 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 278c9fc · merge 278c9fc
 n/a n/a passed #3805 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b7c66ef · merge 7fbd7a3
 n/a n/a passed #4266 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 080f4dd · merge b8a2fa0
 n/a n/a passed #4309 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7dc3871 · merge 7dc3871
 n/a n/a passed #4304 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2acf54c · merge f879b10
 n/a unknown failed #4289 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 1710a25 · merge 507eb19
 n/a provision failed #4290 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
00:59:42 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-04T01:23:08.815Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-04T01:23:11.469Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-04T01:23:12.605Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=ab9828817a590fcc565319d3b9f759ac33a2a189e9aee5854050ce875e1c7bb4
time=2026-03-04T01:25:13.543Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1444352/providers/Microsoft.Resources/deployments/ab9828817a590fcc565319d3b9f759ac33a2a189e9aee5854050ce875e1c7bb4/operationStatuses/08584290194935820161\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1444352/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1444352/providers/Microsoft.Monitor/accounts/hcps-j1444352/providers/Microsoft.Authorization/roleAssignments/c03f7507-bf88-5682-8e3c-44c67e4e4abd\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260304T012327Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'aca8e687-172e-44c1-b3a0-017cb0fe496c', request correlation id 'e5fbf330-df22-4396-b7bd-412e0e32d8a4'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 278c9fc · merge 278c9fc
 n/a n/a passed #3805 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ab14699 · merge ab14699
 n/a n/a passed #3906 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7ec0c2c
 n/a n/a passed #4255 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6420533 · merge 6420533
 n/a n/a passed #4297 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4825dcf · merge ac3354a
 n/a provision failed ⚠#4234 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code CanNotAlterEventHubResourceId; detail message [BadRequest] Cannot alter event hub resource id. Existing: /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.EventHub/namespaces/hcp-prow-westus3-auditlogs-eh/eventhubs/aks-auditlogs, Given: /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.EventHub/namespaces/hcp-prow-westus3-auditlogs-eh/eventhubs/aks-auditlogs; provider Microsoft.EventHub
00:23:54 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/kusto/kustoDataConnection · suite step graph
Show raw failure
time=2026-03-04T01:03:13.616Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=kustoDataConnection description="Step kustoDataConnection\n  Kind: ARM\n  Template: templates/audit-logs-data-connection.bicep\n  Parameters: configurations/audit-logs-data-connection.tmpl.bicepparam"
time=2026-03-04T01:03:14.606Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=kustoDataConnection
time=2026-03-04T01:03:15.530Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=kustoDataConnection deployment=a1c7026655439497a9842d043b8c6ee34a6002875ebdf9d55911262d5cbed8d2
time=2026-03-04T01:03:45.882Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=kusto step=kustoDataConnection err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-kusto-us/providers/Microsoft.Resources/deployments/a1c7026655439497a9842d043b8c6ee34a6002875ebdf9d55911262d5cbed8d2/operationStatuses/08584290206903492434\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"CanNotAlterEventHubResourceId\\\",\\r\\n        \\\"message\\\": \\\"[BadRequest] Cannot alter event hub resource id. Existing: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0241280/providers/Microsoft.EventHub/namespaces/hcp-prow-westus3-auditlogs-eh/eventhubs/aks-auditlogs, Given: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/westus3-shared-resources/providers/Microsoft.EventHub/namespaces/hcp-prow-westus3-auditlogs-eh/eventhubs/aks-auditlogs\\\"\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 723b6e6 · merge 723b6e6
 n/a n/a passed #4287 (merged) 0 n/a