CIHealth
<
Single Day: Apr 2▾
Last 7 Days Weekly: Mar 30 - Apr 5 Sprint: Mar 30 - Apr 12 Single Day: Apr 2
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 22 · Failures matched: 37 · Total runs: 69 · Runs affected: 25

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e913.04%9 of 69 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsMar 27: 1 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 6 · Apr 1: 9 · Apr 2: 10none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009657a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-vdlznq" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-vdlznq\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007d3b30>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009657a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-vdlznq" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-vdlznq\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007d3b30>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a67860>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-46csdr" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-46csdr\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000995eb0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a67860>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-46csdr" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-46csdr\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000995eb0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:206]: Unexpected error:
    <*fmt.wrapErrors | 0xc00090ba70>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-hs6rh7" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-hs6rh7\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010b9110>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:206]: Unexpected error:
    <*fmt.wrapErrors | 0xc00090ba70>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-hs6rh7" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-hs6rh7\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010b9110>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints4
Affected runs (9)
Date (UTC)Associated PRProw job
PR #47202039743836728594432
PR #46882039742819844755456
PR #46872039742386447323136
PR #47422039703704566763520
PR #42102039703492766994432
PR #47002039701224894238720
PR #47232039588637628174336
PR #47022039585436556333056
PR #46302039583088895332352
ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.
full failure pattern:
ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
provision34.35%3 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 1 · Apr 2: 3none
Full failure examples (3)
time=2026-04-02T15:45:11.707Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-04-02T15:45:13.158Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-04-02T15:45:15.083Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=0954c7b8221d8b802b1789f3b78b55cab36ed52110ea91016298121b353b80b4
time=2026-04-02T16:00:18.519Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4296192-svc/providers/Microsoft.Resources/deployments/0954c7b8221d8b802b1789f3b78b55cab36ed52110ea91016298121b353b80b4/operationStatuses/08584264621714434367\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4296192-svc/providers/Microsoft.Resources/deployments/metrics-infra\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-04-02T15:46:48.325Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-04-02T15:46:50.935Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-04-02T15:46:52.408Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=11f5679ed3673e4a9929371cce32c945149645404637979cf67d14076078d9bb
time=2026-04-02T16:02:26.220Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4376448-svc/providers/Microsoft.Resources/deployments/11f5679ed3673e4a9929371cce32c945149645404637979cf67d14076078d9bb/operationStatuses/08584264620738937229\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/aks-svc-kv-access-frontend-cert-prow-j4376448\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/aks-svc-kv-access-admin-api-cert-prow-j4376448\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/aksSPCRead-sessiongate-cert-prow-j4376448\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4376448-svc/providers/Microsoft.Resources/deployments/metrics-infra\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-04-02T15:47:34.702Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-04-02T15:47:35.558Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster
time=2026-04-02T15:47:36.998Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster deployment=a42c05c78c4bc79396dff56a02b6117a0992c06d730dc45d2f0f97c73701fdea
time=2026-04-02T15:59:09.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8920960-mgmt-1/providers/Microsoft.Resources/deployments/a42c05c78c4bc79396dff56a02b6117a0992c06d730dc45d2f0f97c73701fdea/operationStatuses/08584264620293070923\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8920960-mgmt-1/providers/Microsoft.Resources/deployments/cluster-4f5djyc2g2fjc\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/1ddf10d9-e1b0-5dfe-8627-4a983a2a3ab1\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/1ddf10d9-e1b0-5dfe-8627-4a983a2a3ab1\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/d4072e7f-2678-5f20-830e-54226531e4b5\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/d4072e7f-2678-5f20-830e-54226531e4b5\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/388c60cd-66c0-5a3c-bf90-1d1bb63fb700\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/388c60cd-66c0-5a3c-bf90-1d1bb63fb700\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/2c104537-ef93-5756-94dd-a4f0b7a92d1f\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/2c104537-ef93-5756-94dd-a4f0b7a92d1f\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster2
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #46512039724880844296192
PR #39602039724865174376448
PR #46012039724856378920960
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
full failure pattern:
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
provision22.90%2 of 69 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsMar 27: 3 · Mar 28: 0 · Mar 29: 0 · Mar 30: 2 · Mar 31: 3 · Apr 1: 0 · Apr 2: 2none
Full failure examples (2)
time=2026-04-02T19:48:00.107Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-04-02T19:48:02.240Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-04-02T19:48:03.698Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=0c4911c1685a20609549f75db7625cacf2320f36305028898d025569bae2fa4c
time=2026-04-02T19:49:34.390Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2699776/providers/Microsoft.Resources/deployments/0c4911c1685a20609549f75db7625cacf2320f36305028898d025569bae2fa4c/operationStatuses/08584264476026849074\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2699776/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2699776/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j2699776\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report c42af2c3-7d12-4171-9bd0-571348450d45:4/2/2026 7:49:23 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-04-02T09:42:00.307Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-04-02T09:42:02.470Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-04-02T09:42:03.710Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=63e389fe98fd449bd7e9b47c13ebfac931288071ccae0047c27b89b43672b53b
time=2026-04-02T09:45:04.543Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2951808/providers/Microsoft.Resources/deployments/63e389fe98fd449bd7e9b47c13ebfac931288071ccae0047c27b89b43672b53b/operationStatuses/08584264839624194560\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2951808/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2951808/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j2951808\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 899bf80f-8b91-429f-98ec-271a3fd806ee:4/2/2026 9:43:24 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #43932039785313722699776
PR #47052039634347752951808
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...
full failure pattern:
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e22.90%2 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fd3068>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-fgdwpj" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000a93580>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-fgdwpj\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000a93560>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-fgdwpj\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000a93540>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-fgdwpj\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee2c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fd3068>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-fgdwpj" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000a93580>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-fgdwpj\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000a93560>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-fgdwpj\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000a93540>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-fgdwpj\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee2c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012142a0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-s4wxvw" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000ee4ee0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-s4wxvw\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000ee4ec0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-s4wxvw\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000ee4ea0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-s4wxvw\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbedbc60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012142a0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-s4wxvw" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000ee4ee0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-s4wxvw\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000ee4ec0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-s4wxvw\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000ee4ea0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-s4wxvw\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbedbc60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #46522039644149858177024
PR #47302039572528229257216
route host was never found: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
route host was never found: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e22.90%2 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc000db6520>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud on 172.30.0.10:53: no such host
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud on 172.30.0.10:53: no such host",
        err: <*url.Error | 0xc0008de960>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000b10500>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: nil,
                Err: <*net.DNSError | 0xc000b10370>{
                    UnwrapErr: nil,
                    Err: "no such host",
                    Name: "agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud",
                    Server: "172.30.0.10:53",
                    IsTimeout: false,
                    IsTemporary: false,
                    IsNotFound: true,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc000db6520>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud on 172.30.0.10:53: no such host
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud on 172.30.0.10:53: no such host",
        err: <*url.Error | 0xc0008de960>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000b10500>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: nil,
                Err: <*net.DNSError | 0xc000b10370>{
                    UnwrapErr: nil,
                    Err: "no such host",
                    Name: "agnhost-e2e-serving-app-g6d2g.apps.aro.d7t4w3c7c0g3t5g.p107.j5364224.hcp.osadev.cloud",
                    Server: "172.30.0.10:53",
                    IsTimeout: false,
                    IsTemporary: false,
                    IsNotFound: true,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc000e2e4a0>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud on 172.30.0.10:53: no such host
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud on 172.30.0.10:53: no such host",
        err: <*url.Error | 0xc0008d4300>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000b60eb0>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: nil,
                Err: <*net.DNSError | 0xc000a84550>{
                    UnwrapErr: nil,
                    Err: "no such host",
                    Name: "agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud",
                    Server: "172.30.0.10:53",
                    IsTimeout: false,
                    IsTemporary: false,
                    IsNotFound: true,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc000e2e4a0>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud on 172.30.0.10:53: no such host
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud on 172.30.0.10:53: no such host",
        err: <*url.Error | 0xc0008d4300>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000b60eb0>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: nil,
                Err: <*net.DNSError | 0xc000a84550>{
                    UnwrapErr: nil,
                    Err: "no such host",
                    Name: "agnhost-e2e-serving-app-d7kh6.apps.aro.g0h7v0y6q1t2m7r.un2z.j7200896.hcp.osadev.cloud",
                    Server: "172.30.0.10:53",
                    IsTimeout: false,
                    IsTemporary: false,
                    IsNotFound: true,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #46402039712383005364224
PR #47062039710625877200896
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRAll ClusterOperators should report Available=True, but these are not available: [console (Available=) dns (Available=False) image-registry (Available=False) ingress (Available=Fal...
full failure pattern:
All ClusterOperators should report Available=True, but these are not available: [console (Available=) dns (Available=False) image-registry (Available=False) ingress (Available=False) insights (Available=) kube-storage-version-migrator (Available=) monitoring
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.45%1 of 69 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 2 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=) dns (Available=False) image-registry (Available=False) ingress (Available=False) insights (Available=) kube-storage-version-migrator (Available=) monitoring (Available=) openshift-samples (Available=) service-ca (Available=) storage (Available=False)]
Expected
    <[]string | len:10, cap:16>: [
        "console (Available=)",
        "dns (Available=False)",
        "image-registry (Available=False)",
        "ingress (Available=False)",
        "insights (Available=)",
        "kube-storage-version-migrator (Available=)",
        "monitoring (Available=)",
        "openshift-samples (Available=)",
        "service-ca (Available=)",
        "storage (Available=False)",
    ]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=) dns (Available=False) image-registry (Available=False) ingress (Available=False) insights (Available=) kube-storage-version-migrator (Available=) monitoring (Available=) openshift-samples (Available=) service-ca (Available=) storage (Available=False)]
Expected
    <[]string | len:10, cap:16>: [
        "console (Available=)",
        "dns (Available=False)",
        "image-registry (Available=False)",
        "ingress (Available=False)",
        "insights (Available=)",
        "kube-storage-version-migrator (Available=)",
        "monitoring (Available=)",
        "openshift-samples (Available=)",
        "service-ca (Available=)",
        "storage (Available=False)",
    ]
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46882039742819844755456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRAll ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)]
full failure pattern:
All ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)]
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.45%1 of 69 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with:
All ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)]
Expected
    <[]string | len:1, cap:1>: [
        "monitoring (Available=Unknown)",
    ]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with:
All ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)]
Expected
    <[]string | len:1, cap:1>: [
        "monitoring (Available=Unknown)",
    ]
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #47202039743836728594432
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRdeployment router-default -n openshift-ingress has no running pods
full failure pattern:
deployment router-default -n openshift-ingress has no running pods
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.45%1 of 69 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 2 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:180]: Timed out after 600.007s.
router-default deployment logs should be fetchable
Expected success, but got an error:
    <*errors.errorString | 0xc0004bb040>: 
    deployment router-default -n openshift-ingress has no running pods
    {
        s: "deployment router-default -n openshift-ingress has no running pods",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:180]: Timed out after 600.007s.
router-default deployment logs should be fetchable
Expected success, but got an error:
    <*errors.errorString | 0xc0004bb040>: 
    deployment router-default -n openshift-ingress has no running pods
    {
        s: "deployment router-default -n openshift-ingress has no running pods",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46882039742819844755456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
full failure pattern:
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.45%1 of 69 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:186]: Unexpected error:
    <*fmt.wrapError | 0xc0014c4380>: 
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="e2e-cilium-gdtk5x" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"cilium-np\" for cluster \"cilium-cluster\" in resourcegroup=\"e2e-cilium-gdtk5x\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:186]: Unexpected error:
    <*fmt.wrapError | 0xc0014c4380>: 
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="e2e-cilium-gdtk5x" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"cilium-np\" for cluster \"cilium-cluster\" in resourcegroup=\"e2e-cilium-gdtk5x\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46882039742819844755456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRpermission checks failed: expected allowed but denied: list flowcontrol
full failure pattern:
permission checks failed: expected allowed but denied: list flowcontrol
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.45%1 of 69 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc00134a1a0>: 
    permission checks failed: expected allowed but denied: list flowcontrol.apiserver.k8s.io/flowschemas (reason: )
    expected allowed but denied: list flowcontrol.apiserver.k8s.io/prioritylevelconfigurations (reason: )
    expected allowed but denied: list monitoring.coreos.com/prometheusrules (reason: )
    expected allowed but denied: list monitoring.coreos.com/servicemonitors (reason: )
    expected allowed but denied: list machine.openshift.io/machines (reason: )
    expected allowed but denied: list machine.openshift.io/machinesets (reason: )
    expected allowed but denied: list operator.openshift.io/ingresscontrollers (reason: )
    {
        msg: "permission checks failed: expected allowed but denied: list flowcontrol.apiserver.k8s.io/flowschemas (reason: )\nexpected allowed but denied: list flowcontrol.apiserver.k8s.io/prioritylevelconfigurations (reason: )\nexpected allowed but denied: list monitoring.coreos.com/prometheusrules (reason: )\nexpected allowed but denied: list monitoring.coreos.com/servicemonitors (reason: )\nexpected allowed but denied: list machine.openshift.io/machines (reason: )\nexpected allowed but denied: list machine.openshift.io/machinesets (reason: )\nexpected allowed but denied: list operator.openshift.io/ingresscontrollers (reason: )",
        err: <*errors.joinError | 0xc0011e71e8>{
            errs: [
                <*errors.errorString | 0xc0009041c0>{
                    s: "expected allowed but denied: list flowcontrol.apiserver.k8s.io/flowschemas (reason: )",
                },
                <*errors.errorString | 0xc000904390>{
                    s: "expected allowed but denied: list flowcontrol.apiserver.k8s.io/prioritylevelconfigurations (reason: )",
                },
                <*errors.errorString | 0xc0009058a0>{
                    s: "expected allowed but denied: list monitoring.coreos.com/prometheusrules (reason: )",
                },
                <*errors.errorString | 0xc0007b3da0>{
                    s: "expected allowed but denied: list monitoring.coreos.com/servicemonitors (reason: )",
                },
                <*errors.errorString | 0xc000af1e50>{
                    s: "expected allowed but denied: list machine.openshift.io/machines (reason: )",
                },
                <*errors.errorString | 0xc000905d40>{
                    s: "expected allowed but denied: list machine.openshift.io/machinesets (reason: )",
                },
                <*errors.errorString | 0xc0007b3f50>{
                    s: "expected allowed but denied: list operator.openshift.io/ingresscontrollers (reason: )",
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc00134a1a0>: 
    permission checks failed: expected allowed but denied: list flowcontrol.apiserver.k8s.io/flowschemas (reason: )
    expected allowed but denied: list flowcontrol.apiserver.k8s.io/prioritylevelconfigurations (reason: )
    expected allowed but denied: list monitoring.coreos.com/prometheusrules (reason: )
    expected allowed but denied: list monitoring.coreos.com/servicemonitors (reason: )
    expected allowed but denied: list machine.openshift.io/machines (reason: )
    expected allowed but denied: list machine.openshift.io/machinesets (reason: )
    expected allowed but denied: list operator.openshift.io/ingresscontrollers (reason: )
    {
        msg: "permission checks failed: expected allowed but denied: list flowcontrol.apiserver.k8s.io/flowschemas (reason: )\nexpected allowed but denied: list flowcontrol.apiserver.k8s.io/prioritylevelconfigurations (reason: )\nexpected allowed but denied: list monitoring.coreos.com/prometheusrules (reason: )\nexpected allowed but denied: list monitoring.coreos.com/servicemonitors (reason: )\nexpected allowed but denied: list machine.openshift.io/machines (reason: )\nexpected allowed but denied: list machine.openshift.io/machinesets (reason: )\nexpected allowed but denied: list operator.openshift.io/ingresscontrollers (reason: )",
        err: <*errors.joinError | 0xc0011e71e8>{
            errs: [
                <*errors.errorString | 0xc0009041c0>{
                    s: "expected allowed but denied: list flowcontrol.apiserver.k8s.io/flowschemas (reason: )",
                },
                <*errors.errorString | 0xc000904390>{
                    s: "expected allowed but denied: list flowcontrol.apiserver.k8s.io/prioritylevelconfigurations (reason: )",
                },
                <*errors.errorString | 0xc0009058a0>{
                    s: "expected allowed but denied: list monitoring.coreos.com/prometheusrules (reason: )",
                },
                <*errors.errorString | 0xc0007b3da0>{
                    s: "expected allowed but denied: list monitoring.coreos.com/servicemonitors (reason: )",
                },
                <*errors.errorString | 0xc000af1e50>{
                    s: "expected allowed but denied: list machine.openshift.io/machines (reason: )",
                },
                <*errors.errorString | 0xc000905d40>{
                    s: "expected allowed but denied: list machine.openshift.io/machinesets (reason: )",
                },
                <*errors.errorString | 0xc0007b3f50>{
                    s: "expected allowed but denied: list operator.openshift.io/ingresscontrollers (reason: )",
                },
            ],
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #47002039701224894238720
Interrupted by User
full failure pattern:
Interrupted by User
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e11.45%1 of 69 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsMar 27: 4 · Mar 28: 0 · Mar 29: 0 · Mar 30: 12 · Mar 31: 0 · Apr 1: 33 · Apr 2: 1none
Full failure examples (1)
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #47232039744162546323456
Deserializaion Error: no output from command
full failure pattern:
Deserializaion Error: no output from command
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e11.45%1 of 69 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 2 · Mar 31: 0 · Apr 1: 4 · Apr 2: 1none
Full failure examples (1)
HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc0000dbef0 sp=0xc0000dbde8 pc=0x525c458
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc0000dbf78 sp=0xc0000dbef0 pc=0x525ea54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000dbfe0 sp=0xc0000dbf78 pc=0x3af0ab0
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x2dbecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 850
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

goroutine 861 gp=0xc000168000 m=nil [IO wait]:
runtime.gopark(0xc0009da6c0?, 0x3100000000?, 0x50?, 0x7?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000c70720 sp=0xc000c70700 pc=0x2db650e
runtime.netpollblock(0x2e20d18?, 0x2d42066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000c70758 sp=0xc000c70720 pc=0x2d75fb7
internal/poll.runtime_pollWait(0x7f75e94bf600, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000c70778 sp=0xc000c70758 pc=0x2db4da5
internal/poll.(*pollDesc).wait(0xc00050e500?, 0xc00147b000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000c707a0 sp=0xc000c70778 pc=0x2e43a47
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050e500, {0xc00147b000, 0x3000, 0x3000})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000c70838 sp=0xc000c707a0 pc=0x2e44e79
net.(*netFD).Read(0xc00050e500, {0xc00147b000?, 0xf?, 0xc000cd8750?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000c70880 sp=0xc000c70838 pc=0x300a4e5
net.(*conn).Read(0xc00019cb68, {0xc00147b000?, 0x7f75e8391040?, 0x7f76318eaf30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc000c708c8 sp=0xc000c70880 pc=0x301c585
crypto/tls.(*atLeastReader).Read(0xc0010805a0, {0xc00147b000?, 0xc001440b01?, 0xf?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000c70910 sp=0xc000c708c8 pc=0x308801b
bytes.(*Buffer).ReadFrom(0xc00075c9a8, {0x8c14100, 0xc0010805a0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000c70968 sp=0xc000c70910 pc=0x2e81c78
crypto/tls.(*Conn).readFromUntil(0xc00075c708, {0x8c14ee0, 0xc00019cb68}, 0xc000c709d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000c709a0 sp=0xc000c70968 pc=0x30881fe
crypto/tls.(*Conn).readRecordOrCCS(0xc00075c708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000c70c18 sp=0xc000c709a0 pc=0x308527b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc00075c708, {0xc000e62000, 0x1000, 0x34ca980?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000c70c80 sp=0xc000c70c18 pc=0x308bb65
bufio.(*Reader).Read(0xc000fe4600, {0xc000948584, 0x9, 0x34d8a2e?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000c70cb8 sp=0xc000c70c80 pc=0x2f1e1d7
io.ReadAtLeast({0x8c13700, 0xc000fe4600}, {0xc000948584, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000c70d00 sp=0xc000c70cb8 pc=0x2e10e8e
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000948584, 0x9, 0xc00000007b?}, {0x8c13700?, 0xc000fe4600?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000c70d50 sp=0xc000c70d00 pc=0x34c1745
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000c70dd8 sp=0xc000c70d50 pc=0x34c230b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000c70e18 sp=0xc000c70dd8 pc=0x34c2a78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000c70fa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000c70f60 sp=0xc000c70e18 pc=0x34d596a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00097ee00)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000c70fc8 sp=0xc000c70f60 pc=0x34d4cf2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000c70fe0 sp=0xc000c70fc8 pc=0x34ce485
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c70fe8 sp=0xc000c70fe0 pc=0x2dbecc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 860
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2dc0ae3
rdx    0x0
rdi    0xc1bd5b0
rsi    0x80
rbp    0x7ffdf970e738
rsp    0x7ffdf970e6f0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7ffdf970e718
r13    0xc0000cf490
r14    0xc1bb4c0
r15    0xffffffffffffffff
rip    0x2dc0ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc0000dbef0 sp=0xc0000dbde8 pc=0x525c458
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc0000dbf78 sp=0xc0000dbef0 pc=0x525ea54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000dbfe0 sp=0xc0000dbf78 pc=0x3af0ab0
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x2dbecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 850
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

goroutine 861 gp=0xc000168000 m=nil [IO wait]:
runtime.gopark(0xc0009da6c0?, 0x3100000000?, 0x50?, 0x7?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000c70720 sp=0xc000c70700 pc=0x2db650e
runtime.netpollblock(0x2e20d18?, 0x2d42066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000c70758 sp=0xc000c70720 pc=0x2d75fb7
internal/poll.runtime_pollWait(0x7f75e94bf600, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000c70778 sp=0xc000c70758 pc=0x2db4da5
internal/poll.(*pollDesc).wait(0xc00050e500?, 0xc00147b000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000c707a0 sp=0xc000c70778 pc=0x2e43a47
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050e500, {0xc00147b000, 0x3000, 0x3000})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000c70838 sp=0xc000c707a0 pc=0x2e44e79
net.(*netFD).Read(0xc00050e500, {0xc00147b000?, 0xf?, 0xc000cd8750?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000c70880 sp=0xc000c70838 pc=0x300a4e5
net.(*conn).Read(0xc00019cb68, {0xc00147b000?, 0x7f75e8391040?, 0x7f76318eaf30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc000c708c8 sp=0xc000c70880 pc=0x301c585
crypto/tls.(*atLeastReader).Read(0xc0010805a0, {0xc00147b000?, 0xc001440b01?, 0xf?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000c70910 sp=0xc000c708c8 pc=0x308801b
bytes.(*Buffer).ReadFrom(0xc00075c9a8, {0x8c14100, 0xc0010805a0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000c70968 sp=0xc000c70910 pc=0x2e81c78
crypto/tls.(*Conn).readFromUntil(0xc00075c708, {0x8c14ee0, 0xc00019cb68}, 0xc000c709d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000c709a0 sp=0xc000c70968 pc=0x30881fe
crypto/tls.(*Conn).readRecordOrCCS(0xc00075c708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000c70c18 sp=0xc000c709a0 pc=0x308527b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc00075c708, {0xc000e62000, 0x1000, 0x34ca980?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000c70c80 sp=0xc000c70c18 pc=0x308bb65
bufio.(*Reader).Read(0xc000fe4600, {0xc000948584, 0x9, 0x34d8a2e?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000c70cb8 sp=0xc000c70c80 pc=0x2f1e1d7
io.ReadAtLeast({0x8c13700, 0xc000fe4600}, {0xc000948584, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000c70d00 sp=0xc000c70cb8 pc=0x2e10e8e
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000948584, 0x9, 0xc00000007b?}, {0x8c13700?, 0xc000fe4600?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000c70d50 sp=0xc000c70d00 pc=0x34c1745
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000c70dd8 sp=0xc000c70d50 pc=0x34c230b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000c70e18 sp=0xc000c70dd8 pc=0x34c2a78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000c70fa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000c70f60 sp=0xc000c70e18 pc=0x34d596a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00097ee00)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000c70fc8 sp=0xc000c70f60 pc=0x34d4cf2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000c70fe0 sp=0xc000c70fc8 pc=0x34ce485
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c70fe8 sp=0xc000c70fe0 pc=0x2dbecc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 860
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2dc0ae3
rdx    0x0
rdi    0xc1bd5b0
rsi    0x80
rbp    0x7ffdf970e738
rsp    0x7ffdf970e6f0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7ffdf970e718
r13    0xc0000cf490
r14    0xc1bb4c0
r15    0xffffffffffffffff
rip    0x2dc0ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.191
Affected runs (1)
Date (UTC)Associated PRProw job
PR #40042039756884201705472
route host was never found: Get "<url>": dial tcp <ip>:<port>: connect: connection timed out
full failure pattern:
route host was never found: Get "<url>": dial tcp <ip>:<port>: connect: connection timed out
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc00076c140>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-lfmp6.apps.aro.p1p6p1b0v9r5h3k.f9ik.j4755456.hcp.osadev.cloud": dial tcp 4.236.7.70:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-lfmp6.apps.aro.p1p6p1b0v9r5h3k.f9ik.j4755456.hcp.osadev.cloud\": dial tcp 4.236.7.70:443: connect: connection timed out",
        err: <*url.Error | 0xc00173a120>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-lfmp6.apps.aro.p1p6p1b0v9r5h3k.f9ik.j4755456.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc00087a0a0>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc00173a540>{IP: [4, 236, 7, 70], Port: 443, Zone: ""},
                Err: <*os.SyscallError | 0xc00076c080>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc00076c140>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-lfmp6.apps.aro.p1p6p1b0v9r5h3k.f9ik.j4755456.hcp.osadev.cloud": dial tcp 4.236.7.70:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-lfmp6.apps.aro.p1p6p1b0v9r5h3k.f9ik.j4755456.hcp.osadev.cloud\": dial tcp 4.236.7.70:443: connect: connection timed out",
        err: <*url.Error | 0xc00173a120>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-lfmp6.apps.aro.p1p6p1b0v9r5h3k.f9ik.j4755456.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc00087a0a0>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc00173a540>{IP: [4, 236, 7, 70], Port: 443, Zone: ""},
                Err: <*os.SyscallError | 0xc00076c080>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
    <*fmt.wrapError | 0xc0008203a0>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-njnsx.apps.aro.a6o6f3b9m2q2s0q.dcs5.j4755456.hcp.osadev.cloud": dial tcp 20.171.210.37:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-njnsx.apps.aro.a6o6f3b9m2q2s0q.dcs5.j4755456.hcp.osadev.cloud\": dial tcp 20.171.210.37:443: connect: connection timed out",
        err: <*url.Error | 0xc001416000>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-njnsx.apps.aro.a6o6f3b9m2q2s0q.dcs5.j4755456.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000d00140>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc001416150>{
                    IP: [20, 171, 210, 37],
                    Port: 443,
                    Zone: "",
                },
                Err: <*os.SyscallError | 0xc000820040>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
    <*fmt.wrapError | 0xc0008203a0>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-njnsx.apps.aro.a6o6f3b9m2q2s0q.dcs5.j4755456.hcp.osadev.cloud": dial tcp 20.171.210.37:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-njnsx.apps.aro.a6o6f3b9m2q2s0q.dcs5.j4755456.hcp.osadev.cloud\": dial tcp 20.171.210.37:443: connect: connection timed out",
        err: <*url.Error | 0xc001416000>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-njnsx.apps.aro.a6o6f3b9m2q2s0q.dcs5.j4755456.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000d00140>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc001416150>{
                    IP: [20, 171, 210, 37],
                    Port: 443,
                    Zone: "",
                },
                Err: <*os.SyscallError | 0xc000820040>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46882039742819844755456
ERROR CODE: DeploymentNotFound
full failure pattern:
ERROR CODE: DeploymentNotFound
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 5 · Mar 31: 0 · Apr 1: 2 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:86]: Unexpected error:
    <*fmt.wrapError | 0xc00080e300>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-6v7grd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00080e2e0>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc00080e2c0>{
                msg: "failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc000e09d70>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "X-Ms-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Cache": ["CONFIG_NOCACHE"],
                            "Date": [
                                "Thu, 02 Apr 2026 07:54:57 GMT",
                            ],
                            "Pragma": ["no-cache"],
                            "Content-Length": ["112"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                            "Expires": ["-1"],
                            "X-Ms-Routing-Request-Id": [
                                "CANADAEAST:20260402T075458Z:44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "Stri...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:86]: Unexpected error:
    <*fmt.wrapError | 0xc00080e300>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-6v7grd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00080e2e0>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc00080e2c0>{
                msg: "failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc000e09d70>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "X-Ms-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Cache": ["CONFIG_NOCACHE"],
                            "Date": [
                                "Thu, 02 Apr 2026 07:54:57 GMT",
                            ],
                            "Pragma": ["no-cache"],
                            "Content-Length": ["112"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                            "Expires": ["-1"],
                            "X-Ms-Routing-Request-Id": [
                                "CANADAEAST:20260402T075458Z:44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "Stri...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #47052039601679703216128
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 2 · Apr 1: 2 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc001422a98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc000b08900>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc000b088e0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc001422a98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc000b08900>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc000b088e0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46882039742819844755456
failed to create route: the server is currently unable to handle the request
full failure pattern:
failed to create route: the server is currently unable to handle the request
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:179]: Unexpected error:
    <*fmt.wrapError | 0xc000be9020>: 
    failed to create route: the server is currently unable to handle the request
    {
        msg: "failed to create route: the server is currently unable to handle the request",
        err: <*errors.StatusError | 0xc000f014a0>{
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {
                    SelfLink: "",
                    ResourceVersion: "",
                    Continue: "",
                    RemainingItemCount: nil,
                },
                Status: "Failure",
                Message: "the server is currently unable to handle the request",
                Reason: "ServiceUnavailable",
                Details: {
                    Name: "",
                    Group: "",
                    Kind: "",
                    UID: "",
                    Causes: [
                        {
                            Type: "UnexpectedServerResponse",
                            Message: "error trying to reach service: tls: first record does not look like a TLS handshake",
                            Field: "",
                        },
                    ],
                    RetryAfterSeconds: 0,
                },
                Code: 503,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:179]: Unexpected error:
    <*fmt.wrapError | 0xc000be9020>: 
    failed to create route: the server is currently unable to handle the request
    {
        msg: "failed to create route: the server is currently unable to handle the request",
        err: <*errors.StatusError | 0xc000f014a0>{
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {
                    SelfLink: "",
                    ResourceVersion: "",
                    Continue: "",
                    RemainingItemCount: nil,
                },
                Status: "Failure",
                Message: "the server is currently unable to handle the request",
                Reason: "ServiceUnavailable",
                Details: {
                    Name: "",
                    Group: "",
                    Kind: "",
                    UID: "",
                    Causes: [
                        {
                            Type: "UnexpectedServerResponse",
                            Message: "error trying to reach service: tls: first record does not look like a TLS handshake",
                            Field: "",
                        },
                    ],
                    RetryAfterSeconds: 0,
                },
                Code: 503,
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46332039753468280836096
pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range
full failure pattern:
pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc000cc6280>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc000cc6280>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #47382039671529075118080
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc00146d6b0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-ml7xtq, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-ml7xtq" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-ml7xtq, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-ml7xtq\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a074e0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-ml7xtq",
            },
            <*fmt.wrapError | 0xc000972dc0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-ml7xtq\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc00146d6b0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-ml7xtq, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-ml7xtq" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-ml7xtq, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-ml7xtq\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a074e0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-ml7xtq",
            },
            <*fmt.wrapError | 0xc000972dc0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-ml7xtq\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #40042039756884201705472
resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded
full failure pattern:
resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
provision11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 1 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
time=2026-04-02T23:03:56.735Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-04-02T23:03:57.419Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj5102336.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj5102336.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:06e749f2-4dc1-4350-b38e-e21264d3f4fa databaseName:arohcpprow-rp-j5102336 databaseUrl:https://arohcpprow-rp-j5102336.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-9bvn0l27/pipeline@sha256:65f03c580a3645b3375976c0bfefc797ab3ec100b36d5bd2362c81e57a961f14 replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:100m memory:500Mi]] zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-9bvn0l27/pipeline:pull workloadIdentityClientId:5a9df00b-febb-4f42-8e0d-b40467b66a3c workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:06e749f2-4dc1-4350-b38e-e21264d3f4fa workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-04-02T23:03:57.419Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-04-02T23:03:57.479Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-04-02T23:03:57.594Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-04-02T23:03:57.594Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-04-02T23:03:57.706Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-04-02T23:09:02.481Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-04-02T23:09:02.481Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-04-02T23:09:02.481Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-04-02T23:09:05.429Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj5102336.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj5102336.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:06e749f2-4dc1-4350-b38e-e21264d3f4fa databaseName:arohcpprow-rp-j5102336 databaseUrl:https://arohcpprow-rp-j5102336.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-9bvn0l27/pipeline@sha256:65f03c580a3645b3375976c0bfefc797ab3ec100b36d5bd2362c81e57a961f14 replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:100m memory:500Mi]] zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-9bvn0l27/pipeline:pull workloadIdentityClientId:5a9df00b-febb-4f42-8e0d-b40467b66a3c workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:06e749f2-4dc1-4350-b38e-e21264d3f4fa workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-04-02T23:09:05.430Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4xTTW8aMRC9768Y7QmkdYRIP1SqHGjaU5UKtT2l6sFrv4KD17Mae0FE/PjKEMoqWlSub96bmTf2KzwSCSJ3YhDpjqxOOunaY8T1E0z66oKdxSQuLCs6Qt90g38QNggHJLbanODxr4KIqJx3acXinnVyHBbsndmVFZXae96qBkmciarWZo1gDwVhtTJtWR3lP2AEaSG8cRZy73WMmfUiUBGycQZqjZ3a6M6nwRYHztwY7kLqqQe439njkwvWhWV/jIe2EAUPk21cHtKrnEydTA6IPqP1vGsQ0oBugL8AJN8TITmjX21yxbyXJR84uMRy3cx7Dn/c8kG3/Xvo506gpAvJNVDmQLlWe5Gdb5/h/x97bmTReT/wUG3nvarP+ElY/P5YrLsaEpAQv+QPG4s9bVcQUDYRk25aqpG2QKCR1QkZHpXTyfSdmrxRk+nP6e1scjt7+/6xHNPNDV3kfJhNpo/leFzs6YldoLUL9s6FAOnFjAOd09WP1es8FXtqhXP1vGhFLdsj+bomFXleVtQgRr3MLVkshOpdz7xFNH8DAAD//x/fd0cMBAAA"
time=2026-04-02T23:09:05.432Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-04-02T23:09:05.432Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #47452039825828665102336
source "fake-source2.example.com/fake" not found in any IDMS
full failure pattern:
source "fake-source2.example.com/fake" not found in any IDMS
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:210]: Timed out after 600.004s.
both ImageDigestMirrorSet entries should exist on the hosted cluster
Expected success, but got an error:
    <*errors.errorString | 0xc000192db0>: 
    IDMS verification failed:
    source "fake-source2.example.com/fake" not found in any IDMS
    {
        s: "IDMS verification failed:\nsource \"fake-source2.example.com/fake\" not found in any IDMS",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:210]: Timed out after 600.004s.
both ImageDigestMirrorSet entries should exist on the hosted cluster
Expected success, but got an error:
    <*errors.errorString | 0xc000192db0>: 
    IDMS verification failed:
    source "fake-source2.example.com/fake" not found in any IDMS
    {
        s: "IDMS verification failed:\nsource \"fake-source2.example.com/fake\" not found in any IDMS",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #40042039756884201705472
subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
full failure pattern:
subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 2 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.000s.
NFD operator should be installed successfully
Expected success, but got an error:
    <*errors.errorString | 0xc00096ab80>: 
    subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
    {
        s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.000s.
NFD operator should be installed successfully
Expected success, but got an error:
    <*errors.errorString | 0xc00096ab80>: 
    subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
    {
        s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46882039742819844755456
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e11.45%1 of 69 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsMar 27: 0 · Mar 28: 0 · Mar 29: 0 · Mar 30: 0 · Mar 31: 0 · Apr 1: 0 · Apr 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:137]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a32e40>: 
    failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster="np-version-upgrade-cluster-bpwllj" in resourcegroup="rg-np-version-upgrade-bpwllj-plkrh6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009c19a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
            },
            <*fmt.wrapErrors | 0xc000a328a0>{
                msg: "failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009c19a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:137]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a32e40>: 
    failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster="np-version-upgrade-cluster-bpwllj" in resourcegroup="rg-np-version-upgrade-bpwllj-plkrh6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009c19a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
            },
            <*fmt.wrapErrors | 0xc000a328a0>{
                msg: "failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009c19a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #46402039833971700797440