CIHealth
<
Single Day: Mar 10▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 0e576fe · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 240cba2
 n/a n/a passed #4381 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 11ecb87 · merge e0b528c
 n/a provision failed ⚠#4371 (merged) 1
provision (1)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve
22:39:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/global/mirror-blackbox-image · suite step graph
Show raw failure
time=2026-03-10T23:33:28.873Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=global step=mirror-blackbox-image description="Step mirror-blackbox-image\n  Kind: ImageMirror\n  From {<nil> routeMonitorOperator.blackboxExporterImage.registry <nil>}:{<nil> routeMonitorOperator.blackboxExporterImage.repository <nil>}@{<nil> routeMonitorOperator.blackboxExporterImage.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T23:33:53.217Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=global step=mirror-blackbox-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n    sock = connection.create_connection(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n    response = self._make_request(\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n    raise new_e\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n    self._validate_conn(conn)\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n    conn.connect()\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n    self.sock = sock = self._new_conn()\n                       ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n    raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n           ^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n    cmd_result = self.invocation.execute(args)\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n    raise ex\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n    results.append(self._run_job(expanded_arg, cmd_copy))\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n    result = cmd_copy(params)\n             ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n    return self.handler(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n    return op(**command_args)\n           ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n    login_server, _, password = get_login_credentials(\n                                ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n    return _get_credentials(cmd,\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n    return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n                                     ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 276, in _get_aad_token\n    token_params = _handle_challenge_phase(\n                   ^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 100, in _handle_challenge_phase\n    challenge = requests.get(request_url, verify=not should_disable_connection_verify())\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 73, in get\n    return request(\"get\", url, params=params, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n    return session.request(method=method, url=url, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee373fa · merge ee373fa
 n/a n/a passed #4288 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee373fa · merge ee373fa
 n/a unknown failed #4288 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee373fa · merge ee373fa
 n/a unknown failed #4288 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee373fa · merge ee373fa
 n/a unknown failed #4288 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2dcd465
 n/a n/a passed #4393 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head ee373fa · merge ee373fa
 n/a n/a passed #4288 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 66a8da7 · merge 732c175
 n/a provision failed ⚠#4396 (merged) 1
provision (1)
failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors
21:14:29 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-10T21:47:40.879Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-10T21:47:42.919Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-927792105.bicepparam level=Error range=L20:32-L20:33 code=BCP019 err="Expected a new line character at this location."
time=2026-03-10T21:47:42.919Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-927792105.bicepparam level=Error range=L20:32-L20:33 code=BCP019
time=2026-03-10T21:47:42.919Z level=ERROR msg="invalid .bicepparam" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-927792105.bicepparam level=Error range=L21:31-L21:32 code=BCP019 err="Expected a new line character at this location."
time=2026-03-10T21:47:42.919Z level=INFO msg="Expected a new line character at this location." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra source=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/configurations/bicep-params-927792105.bicepparam level=Error range=L21:31-L21:32 code=BCP019
time=2026-03-10T21:47:42.920Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to transform Bicep to ARM: failed to build params: failed to compile bicepparam: 2 errors"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 506b932 · merge 506b932
 n/a n/a passed #4372 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9128c3e · merge 9128c3e
 n/a n/a passed #4383 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c93b50a · merge 8a1ae2a
 n/a n/a passed #4399 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2aab112 · merge 0b33790
 n/a e2e failed #4376 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
20:44:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error:
    <*fmt.wrapError | 0xc00067ed00>: 
    failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-c4qvzv" to finish creating: GET https://rp.j4141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b",
      "name": "350a4761-0c47-48d6-8823-946d38f9fe6b",
      "status": "Failed",
      "startTime": "2026-03-10T21:39:55.404928229Z",
      "endTime": "2026-03-10T21:40:53.386830866Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-c4qvzv\" to finish creating: GET https://rp.j4141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"name\": \"350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T21:39:55.404928229Z\",\n  \"endTime\": \"2026-03-10T21:40:53.386830866Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00067ece0>{
            msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-c4qvzv\" to finish creating: GET https://rp.j4141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"name\": \"350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T21:39:55.404928229Z\",\n  \"endTime\": \"2026-03-10T21:40:53.386830866Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------------------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error:
    <*fmt.wrapError | 0xc00067ed00>: 
    failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-c4qvzv" to finish creating: GET https://rp.j4141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b",
      "name": "350a4761-0c47-48d6-8823-946d38f9fe6b",
      "status": "Failed",
      "startTime": "2026-03-10T21:39:55.404928229Z",
      "endTime": "2026-03-10T21:40:53.386830866Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-c4qvzv\" to finish creating: GET https://rp.j4141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"name\": \"350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T21:39:55.404928229Z\",\n  \"endTime\": \"2026-03-10T21:40:53.386830866Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00067ece0>{
            msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-c4qvzv\" to finish creating: GET https://rp.j4141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"name\": \"350a4761-0c47-48d6-8823-946d38f9fe6b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T21:39:55.404928229Z\",\n  \"endTime\": \"2026-03-10T21:40:53.386830866Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------------------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 06408b4
 n/a n/a passed #4398 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f7dc8e1
 n/a e2e failed #4397 (closed) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
20:27:06 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapError | 0xc000df0dc0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tq24zj" to finish creating: GET https://rp.j1387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41",
      "name": "b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41",
      "status": "Failed",
      "startTime": "2026-03-10T21:43:34.668466002Z",
      "endTime": "2026-03-10T21:44:33.526415167Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-tq24zj\" to finish creating: GET https://rp.j1387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41\",\n  \"name\": \"b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T21:43:34.668466002Z\",\n  \"endTime\": \"2026-03-10T21:44:33.526415167Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0004d7e00>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Request-Id": [
                        "2206b29d-f111-49c3-bab4-1a07c00f8fc4",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 21:44:35 GMT",
                    ],
                    "Content-Length": ["1563"],
                    "X-Envoy-Upstream-Service-Time": ["10"],
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000df0cc0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 98, 56, 49, 48, 51, 98, 99, 100, 45, 52, 100, 99, 50, 45, 52, 99, 51, 102, 45, 56, 97, 52, 52, 45, 98, 51, 102, 102, 54, 101, 56, 48, 54, 100, 52, 49, 34, 44, 10, 32, 32, 32, 32, 34, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapError | 0xc000df0dc0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-tq24zj" to finish creating: GET https://rp.j1387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41",
      "name": "b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41",
      "status": "Failed",
      "startTime": "2026-03-10T21:43:34.668466002Z",
      "endTime": "2026-03-10T21:44:33.526415167Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-tq24zj\" to finish creating: GET https://rp.j1387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41\",\n  \"name\": \"b8103bcd-4dc2-4c3f-8a44-b3ff6e806d41\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T21:43:34.668466002Z\",\n  \"endTime\": \"2026-03-10T21:44:33.526415167Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-193/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0004d7e00>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Request-Id": [
                        "2206b29d-f111-49c3-bab4-1a07c00f8fc4",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 21:44:35 GMT",
                    ],
                    "Content-Length": ["1563"],
                    "X-Envoy-Upstream-Service-Time": ["10"],
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000df0cc0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 98, 56, 49, 48, 51, 98, 99, 100, 45, 52, 100, 99, 50, 45, 52, 99, 51, 102, 45, 56, 97, 52, 52, 45, 98, 51, 102, 102, 54, 101, 56, 48, 54, 100, 52, 49, 34, 44, 10, 32, 32, 32, 32, 34, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 730aedc
 n/a e2e failed ⚠#4157 (open) 21
e2e (21)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000d248b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-p99sfc" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001570c80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-p99sfc\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001570c60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-p99sfc\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001570c40>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-p99sfc\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000d248b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-p99sfc" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001570c80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-p99sfc\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001570c60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-p99sfc\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001570c40>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-p99sfc\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Interrupted by User
20:26:43 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ea6258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2m4dst" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00150ad80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m4dst\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00150ad60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m4dst\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00150ad40>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m4dst\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ea6258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2m4dst" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00150ad80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m4dst\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00150ad60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m4dst\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00150ad40>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m4dst\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
20:26:43 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e1c810>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-negative-tests-gk98xl, error: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-gk98xl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-negative-tests-gk98xl, error: failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-gk98xl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012e4b70>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-negative-tests-gk98xl",
            },
            <*fmt.wrapError | 0xc000aae4c0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-gk98xl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e1c810>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-negative-tests-gk98xl, error: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-gk98xl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-negative-tests-gk98xl, error: failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-gk98xl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012e4b70>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group rg-negative-tests-gk98xl",
            },
            <*fmt.wrapError | 0xc000aae4c0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-gk98xl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ee6138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-fqgjfn" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000a19c20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-fqgjfn\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000a19c00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-fqgjfn\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000a19be0>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-fqgjfn\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ee6138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-fqgjfn" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000a19c20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-fqgjfn\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000a19c00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-fqgjfn\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000a19be0>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-fqgjfn\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ea9398>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-tdzsft" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001038ca0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-tdzsft\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001038c80>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-tdzsft\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001038c60>{
                        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-tdzsft\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ea9398>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-tdzsft" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001038ca0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-tdzsft\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001038c80>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-tdzsft\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001038c60>{
                        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-tdzsft\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
20:26:43 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc001334240>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-22fc9c, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-22fc9c" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-22fc9c, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-22fc9c\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012fbd50>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-22fc9c",
            },
            <*fmt.wrapError | 0xc00136a140>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-22fc9c\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc001334240>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-22fc9c, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-22fc9c" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-22fc9c, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-22fc9c\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012fbd50>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-22fc9c",
            },
            <*fmt.wrapError | 0xc00136a140>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-22fc9c\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
20:26:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc00134ee70>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lqwfph, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-lqwfph" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lqwfph, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lqwfph\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001509300>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lqwfph",
            },
            <*fmt.wrapError | 0xc000e96d80>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lqwfph\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc00134ee70>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lqwfph, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-lqwfph" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lqwfph, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lqwfph\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001509300>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lqwfph",
            },
            <*fmt.wrapError | 0xc000e96d80>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lqwfph\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001406168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-4sswbg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001548da0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4sswbg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001548d80>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4sswbg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001548d60>{
                        msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4sswbg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001406168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-4sswbg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001548da0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4sswbg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001548d80>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4sswbg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001548d60>{
                        msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4sswbg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00020b170>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listqn5zgz" in resourcegroup="ea-list-m4v8xd" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f891a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f89180>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f89160>{
                        msg: "failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00020b170>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listqn5zgz" in resourcegroup="ea-list-m4v8xd" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f891a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f89180>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f89160>{
                        msg: "failed waiting for hcpCluster=\"ea-listqn5zgz\" in resourcegroup=\"ea-list-m4v8xd\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
20:26:43 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:242]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011697a0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-lvqssm, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lvqssm" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-lvqssm, error: failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lvqssm\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000c41360>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-lvqssm",
            },
            <*fmt.wrapError | 0xc000b01060>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lvqssm\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:242]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011697a0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-lvqssm, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lvqssm" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-lvqssm, error: failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lvqssm\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000c41360>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-lvqssm",
            },
            <*fmt.wrapError | 0xc000b01060>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lvqssm\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f7a0a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-97l6pl" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000198900>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-97l6pl\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000198700>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-97l6pl\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00060d060>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-97l6pl\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f7a0a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-97l6pl" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000198900>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-97l6pl\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000198700>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-97l6pl\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00060d060>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-97l6pl\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
20:26:43 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:129]: Unexpected error:
    <*fmt.wrapErrors | 0xc001600cf0>: 
    failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-dt5mrz, error: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-dt5mrz" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-dt5mrz, error: failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-dt5mrz\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014600d0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-dt5mrz",
            },
            <*fmt.wrapError | 0xc001634d60>{
                msg: "failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-dt5mrz\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:129]: Unexpected error:
    <*fmt.wrapErrors | 0xc001600cf0>: 
    failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-dt5mrz, error: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-dt5mrz" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-dt5mrz, error: failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-dt5mrz\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014600d0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-dt5mrz",
            },
            <*fmt.wrapError | 0xc001634d60>{
                msg: "failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-dt5mrz\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Interrupted by User
20:26:43 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded
20:26:43 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010385d0>: 
    failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-bqppvw" for cluster="ea-cluster" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bqppvw, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-bqppvw\" for cluster=\"ea-cluster\" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bqppvw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0000fc8d0>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bqppvw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010385d0>: 
    failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-bqppvw" for cluster="ea-cluster" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bqppvw, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-bqppvw\" for cluster=\"ea-cluster\" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bqppvw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0000fc8d0>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bqppvw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Interrupted by User
20:26:43 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00163a1f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2jqm4s" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007ad920>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2jqm4s\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007ad900>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2jqm4s\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007ad8e0>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2jqm4s\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00163a1f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2jqm4s" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007ad920>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2jqm4s\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007ad900>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2jqm4s\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007ad8e0>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2jqm4s\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0011a41e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-h7jb8n" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000bc7c20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000bc7c00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000bc7be0>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0011a41e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-h7jb8n" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000bc7c20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000bc7c00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000bc7be0>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-h7jb8n\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000da6e88>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dvxxqx" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001c19000>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001c18fe0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001c18fc0>{
                        msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000da6e88>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dvxxqx" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001c19000>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001c18fe0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001c18fc0>{
                        msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-dvxxqx\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010b8318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-glcm9h" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001507800>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-glcm9h\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0015077e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-glcm9h\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0015077c0>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-glcm9h\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010b8318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-glcm9h" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001507800>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-glcm9h\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0015077e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-glcm9h\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0015077c0>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-glcm9h\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
20:26:43 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001824210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-l4vjgt" in resourcegroup="admin-credential-lifecycle-test-gddbct" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00091c8e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l4vjgt\" in resourcegroup=\"admin-credential-lifecycle-test-gddbct\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00091c8c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l4vjgt\" in resourcegroup=\"admin-credential-lifecycle-test-gddbct\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00091c8a0>{
                        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-l4vjgt\" in resourcegroup=\"admin-credential-lifecycle-test-gddbct\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001824210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-l4vjgt" in resourcegroup="admin-credential-lifecycle-test-gddbct" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00091c8e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l4vjgt\" in resourcegroup=\"admin-credential-lifecycle-test-gddbct\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00091c8c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l4vjgt\" in resourcegroup=\"admin-credential-lifecycle-test-gddbct\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00091c8a0>{
                        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-l4vjgt\" in resourcegroup=\"admin-credential-lifecycle-test-gddbct\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5a9cad4 · merge 5a9cad4
 n/a n/a passed #4395 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1b8d093
 n/a e2e failed #4198 (open) 12
e2e (12)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000eb6528>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-7smhwr" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007e3ca0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-7smhwr\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007e3c80>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-7smhwr\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007e3c60>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-7smhwr\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000eb6528>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-7smhwr" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007e3ca0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-7smhwr\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007e3c80>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-7smhwr\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007e3c60>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-7smhwr\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001076768>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-khhpxr" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d2c5e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-khhpxr\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d2c5c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-khhpxr\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d2c5a0>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-khhpxr\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001076768>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-khhpxr" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d2c5e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-khhpxr\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d2c5c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-khhpxr\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d2c5a0>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-khhpxr\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001792210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listrckpfl" in resourcegroup="ea-list-tqbdp4" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009e8cc0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listrckpfl\" in resourcegroup=\"ea-list-tqbdp4\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009e8ca0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listrckpfl\" in resourcegroup=\"ea-list-tqbdp4\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009e8a80>{
                        msg: "failed waiting for hcpCluster=\"ea-listrckpfl\" in resourcegroup=\"ea-list-tqbdp4\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001792210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listrckpfl" in resourcegroup="ea-list-tqbdp4" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009e8cc0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listrckpfl\" in resourcegroup=\"ea-list-tqbdp4\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009e8ca0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listrckpfl\" in resourcegroup=\"ea-list-tqbdp4\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009e8a80>{
                        msg: "failed waiting for hcpCluster=\"ea-listrckpfl\" in resourcegroup=\"ea-list-tqbdp4\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e942e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-shn2vs" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00096f540>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-shn2vs\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00096f520>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-shn2vs\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00096f500>{
                        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-shn2vs\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e942e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-shn2vs" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00096f540>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-shn2vs\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00096f520>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-shn2vs\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00096f500>{
                        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-shn2vs\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000d46240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-pqbh6w" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0006fed60>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pqbh6w\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0006fed40>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pqbh6w\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0006fed20>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pqbh6w\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000d46240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-pqbh6w" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0006fed60>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pqbh6w\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0006fed40>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pqbh6w\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0006fed20>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pqbh6w\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00151a150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m5vhcc" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000801020>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m5vhcc\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000801000>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m5vhcc\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000800fe0>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m5vhcc\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00151a150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m5vhcc" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000801020>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m5vhcc\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000801000>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m5vhcc\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000800fe0>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m5vhcc\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000efe468>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-tdcw2z" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001015680>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tdcw2z\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001015660>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tdcw2z\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001015640>{
                        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tdcw2z\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000efe468>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-tdcw2z" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001015680>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tdcw2z\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001015660>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tdcw2z\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001015640>{
                        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tdcw2z\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00088a9d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6cxnq5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009011c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6cxnq5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009011a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6cxnq5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000901180>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6cxnq5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00088a9d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6cxnq5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009011c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6cxnq5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009011a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6cxnq5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000901180>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6cxnq5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
19:51:46 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:152]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014d4810>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-6ggxsk" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-6ggxsk, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-6ggxsk\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-6ggxsk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00094e200>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-6ggxsk",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:152]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014d4810>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-6ggxsk" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-6ggxsk, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-6ggxsk\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-6ggxsk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00094e200>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-6ggxsk",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00134c1e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-5t96v4" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000b19140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5t96v4\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000b19120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5t96v4\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000b19100>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5t96v4\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00134c1e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-5t96v4" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000b19140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5t96v4\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000b19120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5t96v4\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000b19100>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5t96v4\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ef4648>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-tl4jh6" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f221c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-tl4jh6\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f221a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-tl4jh6\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f22180>{
                        msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-tl4jh6\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ef4648>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-tl4jh6" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f221c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-tl4jh6\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f221a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-tl4jh6\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f22180>{
                        msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-tl4jh6\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
19:51:46 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000eb6120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-gn8rhl" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00072d380>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-gn8rhl\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00072d340>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-gn8rhl\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00072d300>{
                        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-gn8rhl\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000eb6120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-gn8rhl" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00072d380>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-gn8rhl\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00072d340>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-gn8rhl\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00072d300>{
                        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-gn8rhl\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0650c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2aea2a8 · merge 8550ece
 n/a n/a passed #4219 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 3aeb6fa
 n/a n/a passed #4394 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9505bdb
 n/a provision failed ⚠#4381 (closed) 1
provision (1)
error: reading JSON file "/tmp/oc-auth-3352314194.json": unmarshaling JSON at "/tmp/oc-auth-3352314194.json": unexpected end of JSON input
19:04:56 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/global/mirror-image · suite step graph
Show raw failure
time=2026-03-10T19:55:34.522Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T19:55:34.846Z level=INFO msg="Source registry is a CI registry, using oc registry login" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image registry=registry.build02.ci.openshift.org
time=2026-03-10T19:55:34.995Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to get CI registry credentials via oc: oc registry login failed: info: Using registry public hostname registry.build02.ci.openshift.org\nerror: reading JSON file \"/tmp/oc-auth-3352314194.json\": unmarshaling JSON at \"/tmp/oc-auth-3352314194.json\": unexpected end of JSON input\n: exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 1892906 · merge 1892906
 n/a n/a passed #3820 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1c23be7
 n/a provision failed ⚠#4157 (open) 1
provision (1)
failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded
18:36:26 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph
Show raw failure
time=2026-03-10T19:13:32.990Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-10T19:13:36.046Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-10T19:13:37.593Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=ae8370847c9f196727dbb8b739d24301b9609659ed95d4a27f00945ecff16f8f
time=2026-03-10T19:43:32.726Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee373fa · merge ee373fa
 n/a n/a passed #4288 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9128c3e · merge 9128c3e
 n/a e2e failed ★#4383 (merged) 5
e2e (5)
ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift
18:33:45 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc000976620>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-77cmhg" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
      "name": "9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
      "status": "Failed",
      "startTime": "2026-03-10T19:27:23.96442781Z",
      "endTime": "2026-03-10T19:29:08.437592376Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\nTime:2026-03-10T19:28:51.1177193Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n  \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000976600>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n  \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.R...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc000976620>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-77cmhg" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
      "name": "9c94ce7e-df55-4d68-8fe3-b0f80110cfea",
      "status": "Failed",
      "startTime": "2026-03-10T19:27:23.96442781Z",
      "endTime": "2026-03-10T19:29:08.437592376Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\nTime:2026-03-10T19:28:51.1177193Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n  \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000976600>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-77cmhg\" to finish creating: GET https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"name\": \"9c94ce7e-df55-4d68-8fe3-b0f80110cfea\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:27:23.96442781Z\",\n  \"endTime\": \"2026-03-10T19:29:08.437592376Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj3677056.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f8cd3561-101e-0059-4fc4-b09be1000000\\nTime:2026-03-10T19:28:51.1177193Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.R...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error
18:33:45 UTC · failed at e2e · test Engineering should be able to retrieve expected metrics from the /metrics endpoint · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc0000df9e0>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc000746350>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc0000df9e0>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc000746350>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.7:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift
18:33:45 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapError | 0xc0012ab180>: 
    failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000fafb60>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Date": [
                        "Tue, 10 Mar 2026 19:34:29 GMT",
                    ],
                    "Content-Length": ["316"],
                    "X-Envoy-Upstream-Service-Time": ["25"],
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "f582fe12-33f9-4c9a-95d4-c934cb1441c8",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0012ab0a0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 118, 114, 106, 99, 122, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 316,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j3677056.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredenti...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapError | 0xc0012ab180>: 
    failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000fafb60>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Date": [
                        "Tue, 10 Mar 2026 19:34:29 GMT",
                    ],
                    "Content-Length": ["316"],
                    "X-Envoy-Upstream-Service-Time": ["25"],
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "f582fe12-33f9-4c9a-95d4-c934cb1441c8",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0012ab0a0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 118, 114, 106, 99, 122, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 112, 97, 116, 99, 104, 45, 116, 97, 103, 115, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 316,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j3677056.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vrjcz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredenti...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift
18:33:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapError | 0xc000a02c20>: 
    failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000f47e60>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "82c53113-ce1b-4dd4-bd27-dd5b5ee92864",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 19:34:24 GMT",
                    ],
                    "Content-Length": ["346"],
                    "X-Envoy-Upstream-Service-Time": ["27"],
                    "Server": ["istio-envoy"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000a02a80>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 122, 98, 116, 50, 109, 108, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 346,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j3677056.hcpsvc.osadev.cloud",
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapError | 0xc000a02c20>: 
    failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-zbt2ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000f47e60>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "82c53113-ce1b-4dd4-bd27-dd5b5ee92864",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 19:34:24 GMT",
                    ],
                    "Content-Length": ["346"],
                    "X-Envoy-Upstream-Service-Time": ["27"],
                    "Server": ["istio-envoy"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000a02a80>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 122, 98, 116, 50, 109, 108, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 100, 105, 115, 97, 98, 108, 101, 100, 45, 105, 109, 97, 103, 101, 45, 114, 101, 103, 105, 115, 116, 114, 121, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 346,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j3677056.hcpsvc.osadev.cloud",
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift
18:33:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapError | 0xc000a0b220>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4xs98c": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a0b200>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000990c90>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "d89e5839-2ec1-4aaa-a7fe-70f6904cc07d",
                        ],
                        "Date": [
                            "Tue, 10 Mar 2026 19:34:28 GMT",
                        ],
                        "Content-Length": ["340"],
                        "X-Envoy-Upstream-Service-Time": ["2101"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a0b120>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapError | 0xc000a0b220>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4xs98c": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a0b200>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4xs98c\": PUT https://rp.j3677056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-4xs98c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000990c90>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "d89e5839-2ec1-4aaa-a7fe-70f6904cc07d",
                        ],
                        "Date": [
                            "Tue, 10 Mar 2026 19:34:28 GMT",
                        ],
                        "Content-Length": ["340"],
                        "X-Envoy-Upstream-Service-Time": ["2101"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a0b120>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 7ebfd60 · merge 04b8269
 n/a provision failed #4392 (merged) 1
provision (1)
resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded
18:26:00 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy · suite step graph
Show raw failure
time=2026-03-10T19:20:11.600Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-10T19:20:12.198Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8216448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8216448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:066a9a53-0fdb-46be-ac16-f32bfa2af0ac databaseName:arohcpprow-rp-j8216448 databaseUrl:https://arohcpprow-rp-j8216448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-6tqc6hj7/pipeline@sha256:3f5bf8fef96269f6b0b7fe4d229a38a31b921ec076789d23b904156eb1abb94d replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6tqc6hj7/pipeline:pull workloadIdentityClientId:c0b7e765-ba72-4ade-ba4f-7eb7f1c73b5f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:066a9a53-0fdb-46be-ac16-f32bfa2af0ac workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-10T19:20:12.198Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-10T19:20:12.441Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-10T19:20:12.740Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-03-10T19:20:12.740Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-10T19:20:12.850Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-10T19:25:19.095Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-10T19:25:19.095Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-10T19:25:19.095Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-10T19:25:22.577Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8216448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8216448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:066a9a53-0fdb-46be-ac16-f32bfa2af0ac databaseName:arohcpprow-rp-j8216448 databaseUrl:https://arohcpprow-rp-j8216448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-6tqc6hj7/pipeline@sha256:3f5bf8fef96269f6b0b7fe4d229a38a31b921ec076789d23b904156eb1abb94d replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6tqc6hj7/pipeline:pull workloadIdentityClientId:c0b7e765-ba72-4ade-ba4f-7eb7f1c73b5f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:066a9a53-0fdb-46be-ac16-f32bfa2af0ac workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-10T19:25:22.579Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTQW8aMRCF7/srRnsCaR0BVSuFKoc07alKhZqeUvXgtV/BwetZjb0gIn58ZWjKFkHC9e1749nPfoVHIkHkTgwi3ZDVSSddewy4foJJX12w05jEhXlFe+mbbvBPwgphp8RWmxd5+LMgIiofICtncM/BJZayolILq4VpVa3NEsH2pLL6L3NrDHchZcN573f2yLKHthAFD5MchzPOTy5YF+a9kert4N91TuyuGiRxJp4I3XH47eb3uu2fpZ87gZIuJNdAmZ3l0uxZ92e0njcN9qTepntrZNZ5fwJF23mv6oN+HOzSgsU968xpxt6Zzc7lPa9fSLxy7gyQPAIhOaOPWF9A9AFGkGbCK2chd17H2F8+7m9JLbFRK935Po2y+PWxWHY1JCAhfskPNhZbWi8goHwZMemmpRppDQQaWJ2Q5UE5GU0+qNE7NR79GF9PJ6PpePJYDunqis563k/H14/lcFhs6YldoKUL9saFAOnVjAMd2tWv1XGfii21wvnrYdGKWrZ782VDKvI8r6hBjHqeR7JYCNWb3s9bRPMnAAD//9ypvxEMBAAA"
time=2026-03-10T19:25:22.582Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-10T19:25:22.582Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 506b932 · merge 506b932
 n/a e2e failed #4372 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift
18:23:33 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc0009d0140>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-96bfsn" to finish creating: GET https://rp.j4466944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632",
      "name": "1171ec57-e6db-40d0-a6ce-6dbca99f9632",
      "status": "Failed",
      "startTime": "2026-03-10T19:50:09.79918139Z",
      "endTime": "2026-03-10T19:53:08.23159911Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj4466944.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:2e42c44b-501e-0099-1dc7-b0ee07000000\nTime:2026-03-10T19:52:47.7867575Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-96bfsn\" to finish creating: GET https://rp.j4466944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"name\": \"1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:50:09.79918139Z\",\n  \"endTime\": \"2026-03-10T19:53:08.23159911Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj4466944.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:2e42c44b-501e-0099-1dc7-b0ee07000000\\nTime:2026-03-10T19:52:47.7867575Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009d0120>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-96bfsn\" to finish creating: GET https://rp.j4466944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"name\": \"1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:50:09.79918139Z\",\n  \"endTime\": \"2026-03-10T19:53:08.23159911Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj4466944.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:2e42c44b-501e-0099-1dc7-b0ee07000000\\nTime:2026-03-10T19:52:47.7867575Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc0009d0140>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-96bfsn" to finish creating: GET https://rp.j4466944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632",
      "name": "1171ec57-e6db-40d0-a6ce-6dbca99f9632",
      "status": "Failed",
      "startTime": "2026-03-10T19:50:09.79918139Z",
      "endTime": "2026-03-10T19:53:08.23159911Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj4466944.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:2e42c44b-501e-0099-1dc7-b0ee07000000\nTime:2026-03-10T19:52:47.7867575Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-96bfsn\" to finish creating: GET https://rp.j4466944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"name\": \"1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:50:09.79918139Z\",\n  \"endTime\": \"2026-03-10T19:53:08.23159911Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj4466944.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:2e42c44b-501e-0099-1dc7-b0ee07000000\\nTime:2026-03-10T19:52:47.7867575Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009d0120>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-96bfsn\" to finish creating: GET https://rp.j4466944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"name\": \"1171ec57-e6db-40d0-a6ce-6dbca99f9632\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T19:50:09.79918139Z\",\n  \"endTime\": \"2026-03-10T19:53:08.23159911Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj4466944.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:2e42c44b-501e-0099-1dc7-b0ee07000000\\nTime:2026-03-10T19:52:47.7867575Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dacf876
 n/a unknown failed #4391 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b136b7d · merge 138964d
 n/a e2e failed #4226 (merged) 1
e2e (1)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
18:07:48 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc0000269c0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-mcjzx6" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mcjzx6, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-mcjzx6\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mcjzx6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a55ad0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mcjzx6",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc0000269c0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-mcjzx6" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mcjzx6, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-mcjzx6\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mcjzx6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a55ad0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-mcjzx6",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 244d96c · merge 244d96c
 n/a n/a passed #4319 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 830c521
 n/a n/a passed #4305 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4487e54 · merge e9ca15d
 n/a n/a passed #3907 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head e430bc4 · merge a36f37a
 n/a n/a passed #4205 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 1892906 · merge 1892906
 n/a provision failed #3820 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
16:39:45 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-10T17:09:20.033Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-10T17:09:22.741Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-10T17:09:24.001Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=252e7d91bcfd9798a43b366e8b47cccab07d0ba0b87fa05438e88bf174eb66d9
time=2026-03-10T17:10:54.772Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4835840/providers/Microsoft.Resources/deployments/252e7d91bcfd9798a43b366e8b47cccab07d0ba0b87fa05438e88bf174eb66d9/operationStatuses/08584284443221733016\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4835840/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4835840/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j4835840\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report ac344776-a670-42b1-bee8-cfda1e896ad5:3/10/2026 5:10:44 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head dca081b · merge a82c625
 n/a n/a passed #4347 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head e680d6c · merge 506b932
 n/a unknown failed #4372 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 69b2b21 · merge 69b2b21
 n/a n/a passed #4380 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee373fa · merge ee373fa
 n/a unknown failed #4288 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head ee373fa · merge ee373fa
 n/a provision failed #4288 (merged) 1
provision (1)
error running Shell Step, failed to execute shell command: + set -euo pipefail + ISTIO_NAMESPACE=aks-istio-system + echo '********** Check istio is up and running **************' ********** Check istio is up and running ************** ++ kubectl get pod
15:42:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade · suite step graph
Show raw failure
time=2026-03-10T16:33:07.282Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: Shell\n  Command: ./istio.sh\n"
time=2026-03-10T16:33:09.573Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="error running Shell Step, failed to execute shell command: + set -euo pipefail\n+ ISTIO_NAMESPACE=aks-istio-system\n+ echo '********** Check istio is up and running **************'\n********** Check istio is up and running **************\n++ kubectl get pods -n aks-istio-system -l istio.io/rev=asm-1-26 --field-selector=status.phase=Running --no-headers\n++ wc -l\n+ ISTIO_PODS_COUNT=2\n+ [[ 2 -lt 2 ]]\n+ echo '********** Download istioctl **************'\n********** Download istioctl **************\n++ uname\n+ OS=Linux\n+ '[' Linux = Darwin ']'\n+ OSEXT=linux\n++ uname -m\n+ LOCAL_ARCH=x86_64\n+ case \"${LOCAL_ARCH}\" in\n+ ISTIO_ARCH=amd64\n+ ISTIO_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz\n+ SHA256_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz -o istio-1.28.3-linux-amd64.tar.gz\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256 -o istio-1.28.3-linux-amd64.tar.gz.sha256\n+ sha256sum -c istio-1.28.3-linux-amd64.tar.gz.sha256\nsha256sum: istio-1.28.3-linux-amd64.tar.gz.sha256: no properly formatted SHA256 checksum lines found\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4487e54 · merge e9ca15d
 n/a provision failed ⚠#3907 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code BadRequest; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/admin-api' was not found.; provider Microsoft.ManagedIdentity
15:42:10 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph
Show raw failure
time=2026-03-10T16:19:05.157Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-10T16:19:06.686Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-10T16:19:08.195Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=193466c63e88ab1f67c893b50981287f131a4d95cda9d0c1bf16aaa9bbd6a676
time=2026-03-10T16:32:41.387Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4114176-svc/providers/Microsoft.Resources/deployments/193466c63e88ab1f67c893b50981287f131a4d95cda9d0c1bf16aaa9bbd6a676/operationStatuses/08584284473381261118\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4114176/providers/Microsoft.Resources/deployments/rp_cosmos_db\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"FailedIdentityOperation\\\",\\r\\n            \\\"message\\\": \\\"Identity operation for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4114176/providers/Microsoft.DocumentDB/databaseAccounts/arohcpprow-rp-j4114176' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\\\\\\\"error\\\\\\\":{\\\\\\\"code\\\\\\\":\\\\\\\"BadRequest\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4114176-svc/providers/Microsoft.ManagedIdentity/userAssignedIdentities/admin-api' was not found.\\\\\\\"}}'.'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 11ecb87 · merge e0b528c
 n/a e2e failed ★#4371 (merged) 1
e2e (1)
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error
15:27:56 UTC · failed at e2e · test Engineering should be able to retrieve expected metrics from the /metrics endpoint · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc000894420>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc00069a270>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc000894420>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc00069a270>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.11:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head e884774 · merge e884774
 n/a unknown failed #4388 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 6cb8a51 · merge 9128c3e
 n/a n/a passed #4383 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 4d70afb
 n/a e2e failed ⚠#4305 (closed) 5
e2e (5)
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift
14:42:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapError | 0xc000b751e0>: 
    failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00133aff0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Date": [
                        "Tue, 10 Mar 2026 15:57:57 GMT",
                    ],
                    "Content-Length": ["327"],
                    "X-Envoy-Upstream-Service-Time": ["27"],
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "eba75de9-6f77-49a9-8261-41301d1686fe",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000b750a0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 114, 103, 45, 99, 108, 117, 115, 116, 101, 114, 45, 98, 97, 99, 107, 45, 118, 101, 114, 115, 105, 111, 110, 45, 53, 119, 104, 113, 102, 54, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 99, 108, 117, 115, 116, 101, 114, 45, 118, 101, 114, 45, 52, 45, 49, 57, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 327,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j1986688.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapError | 0xc000b751e0>: 
    failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00133aff0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Date": [
                        "Tue, 10 Mar 2026 15:57:57 GMT",
                    ],
                    "Content-Length": ["327"],
                    "X-Envoy-Upstream-Service-Time": ["27"],
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "eba75de9-6f77-49a9-8261-41301d1686fe",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000b750a0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 114, 103, 45, 99, 108, 117, 115, 116, 101, 114, 45, 98, 97, 99, 107, 45, 118, 101, 114, 115, 105, 111, 110, 45, 53, 119, 104, 113, 102, 54, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 99, 108, 117, 115, 116, 101, 114, 45, 118, 101, 114, 45, 52, 45, 49, 57, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 327,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j1986688.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-5whqf6/providers/...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift
14:42:05 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:173]: Unexpected error:
    <*fmt.wrapError | 0xc00141e7a0>: 
    failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0008b7830>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "032dd7ef-e3c7-4688-a613-f8a1b5d25b8e",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 15:57:48 GMT",
                    ],
                    "Content-Length": ["346"],
                    "X-Envoy-Upstream-Service-Time": ["1028"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00141e6a0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 101, 115, 116, 45, 104, 120, 106, 107, 56, 109, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 57, 53, 54, 104, 102, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 346,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j1986688.hcpsvc.osadev.cloud",
       ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:173]: Unexpected error:
    <*fmt.wrapError | 0xc00141e7a0>: 
    failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hxjk8m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-t956hf\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0008b7830>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "032dd7ef-e3c7-4688-a613-f8a1b5d25b8e",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 15:57:48 GMT",
                    ],
                    "Content-Length": ["346"],
                    "X-Envoy-Upstream-Service-Time": ["1028"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00141e6a0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 101, 115, 116, 45, 104, 120, 106, 107, 56, 109, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 97, 100, 109, 105, 110, 45, 99, 114, 101, 100, 45, 108, 105, 102, 101, 99, 121, 99, 108, 101, 45, 116, 57, 53, 54, 104, 102, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 346,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j1986688.hcpsvc.osadev.cloud",
       ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
source "fake-source2.example.com/fake" not found in any IDMS
14:42:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:214]: Timed out after 300.003s.
both ImageDigestMirrorSet entries should exist on the hosted cluster
Expected success, but got an error:
    <*errors.errorString | 0xc000b170c0>: 
    IDMS verification failed:
    source "fake-source2.example.com/fake" not found in any IDMS
    {
        s: "IDMS verification failed:\nsource \"fake-source2.example.com/fake\" not found in any IDMS",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:214]: Timed out after 300.003s.
both ImageDigestMirrorSet entries should exist on the hosted cluster
Expected success, but got an error:
    <*errors.errorString | 0xc000b170c0>: 
    IDMS verification failed:
    source "fake-source2.example.com/fake" not found in any IDMS
    {
        s: "IDMS verification failed:\nsource \"fake-source2.example.com/fake\" not found in any IDMS",
    }
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift
14:42:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc000775300>: 
    failed to create NodePool nodepool-128GiB: failed starting nodepool creation "nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-g7rf7z": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool nodepool-128GiB: failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007752e0>{
            msg: "failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011b5da0>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Date": [
                            "Tue, 10 Mar 2026 15:57:57 GMT",
                        ],
                        "Content-Length": ["344"],
                        "X-Envoy-Upstream-Service-Time": ["85"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "7e936490-be89-47eb-a302-7b10d5507757",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000775200>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc000775300>: 
    failed to create NodePool nodepool-128GiB: failed starting nodepool creation "nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-g7rf7z": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool nodepool-128GiB: failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007752e0>{
            msg: "failed starting nodepool creation \"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-g7rf7z\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-g7rf7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/nodePools/nodepool-128GiB\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011b5da0>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Date": [
                            "Tue, 10 Mar 2026 15:57:57 GMT",
                        ],
                        "Content-Length": ["344"],
                        "X-Envoy-Upstream-Service-Time": ["85"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "7e936490-be89-47eb-a302-7b10d5507757",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000775200>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift
14:42:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapError | 0xc000a3afe0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-g779ch": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a3afc0>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012bad50>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "3707dcaf-af3a-4d6e-bc43-be2c2e5217f8",
                        ],
                        "Date": [
                            "Tue, 10 Mar 2026 15:57:56 GMT",
                        ],
                        "Content-Length": ["345"],
                        "X-Envoy-Upstream-Service-Time": ["103"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a3aee0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapError | 0xc000a3afe0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-g779ch": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a3afc0>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-g779ch\": PUT https://rp.j1986688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g779ch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012bad50>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "3707dcaf-af3a-4d6e-bc43-be2c2e5217f8",
                        ],
                        "Date": [
                            "Tue, 10 Mar 2026 15:57:56 GMT",
                        ],
                        "Content-Length": ["345"],
                        "X-Envoy-Upstream-Service-Time": ["103"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000a3aee0>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head eac14df · merge 131806c
 n/a e2e failed #4351 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
14:09:07 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error:
    <*errors.joinError | 0xc001362360>: 
    cluster listing in non-existent resource group: expected error but none occurred
    node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*errors.errorString | 0xc001142210>{
                s: "cluster listing in non-existent resource group: expected error but none occurred",
            },
            <*errors.errorString | 0xc0011431a0>{
                s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
            <*errors.errorString | 0xc000f366a0>{
                s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:244]: Unexpected error:
    <*errors.joinError | 0xc001362360>: 
    cluster listing in non-existent resource group: expected error but none occurred
    node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*errors.errorString | 0xc001142210>{
                s: "cluster listing in non-existent resource group: expected error but none occurred",
            },
            <*errors.errorString | 0xc0011431a0>{
                s: "node pool listing in RG with no clusters: expected error containing keywords 'parent resource not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jjgrg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
            <*errors.errorString | 0xc000f366a0>{
                s: "node pool listing in non-existent resource group: expected error containing keywords 'resource group not found', got: GET https://rp.j9416064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/non-existent-rg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head eac14df · merge 131806c
 n/a unknown failed #4351 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1c23be7
 n/a e2e failed ⚠#4157 (open) 21
e2e (21)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0000fe228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-k726vg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d9b820>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-k726vg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d9b800>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-k726vg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d9b7e0>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-k726vg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0000fe228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-k726vg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d9b820>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-k726vg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d9b800>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-k726vg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d9b7e0>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-k726vg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
13:59:50 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:129]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011b9740>: 
    failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-lq8w22, error: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-lq8w22" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-lq8w22, error: failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-lq8w22\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000166850>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-lq8w22",
            },
            <*fmt.wrapError | 0xc001000ee0>{
                msg: "failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-lq8w22\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:129]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011b9740>: 
    failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-lq8w22, error: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-lq8w22" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-lq8w22, error: failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-lq8w22\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000166850>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-lq8w22",
            },
            <*fmt.wrapError | 0xc001000ee0>{
                msg: "failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-lq8w22\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fdc2b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-sq6hmg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0012e8fe0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-sq6hmg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0012e8fc0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-sq6hmg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0012e8fa0>{
                        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-sq6hmg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fdc2b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-sq6hmg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0012e8fe0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-sq6hmg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0012e8fc0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-sq6hmg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0012e8fa0>{
                        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-sq6hmg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e82408>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wdq7k7" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00052b6a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wdq7k7\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00052b680>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wdq7k7\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00052b660>{
                        msg: "failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wdq7k7\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e82408>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wdq7k7" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00052b6a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wdq7k7\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00052b680>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wdq7k7\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00052b660>{
                        msg: "failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wdq7k7\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Deserializaion Error: no output from command
13:59:50 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
s report:
      Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled (Spec Runtime: 1h29m59.624s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.623s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
          At [By Step] creating the HCP cluster (Step Runtime: 36m8.55s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:76

          Spec Goroutine
          goroutine 37 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc0012e8460}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc0012e8460}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0001aca80?}, {{0x824f9c8?, 0xc0004c0000?}, 0xc00063f340?}, 0xc00009b7e0, {0xc0014343c0, 0x15}, {0x7841a74, 0x14}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func23.2({0x7fbcf0684460, 0xc00066a2a0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:77
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc00066a2a0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 81
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83 @ 03/10/26 16:42:11.169
  "ts"="2026-03-10 16:42:11.169810" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.170007" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-5k975s" "resourceGroup"="np-autoscaling-cb9l84"
  "ts"="2026-03-10 16:42:12.605524" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-5k975s" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.934752" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-10 16:42:13.934928" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-10 16:42:13.935067" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.938167" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-146"
  "ts"="2026-03-10 16:42:13.938214" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-cb9l84"
  "ts"="2026-03-10 16:42:14.762169" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-146" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:15.414810" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-cb9l84" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
  "ts"="2026-03-10 16:42:16.145633" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-146" "deployment"="msi-deployment-146"
  "ts"="2026-03-10 16:42:16.860089" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-cb9l84" "deployment"="customer-infra-np-autoscale-cluster-5k975s"
  "ts"="2026-03-10 16:42:18.287841" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.288071" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.288315" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-cb9l84"
Command Error: signal: killed
Deserializaion Error: no output from command
s report:
      Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled (Spec Runtime: 1h29m59.624s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.623s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
          At [By Step] creating the HCP cluster (Step Runtime: 36m8.55s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:76

          Spec Goroutine
          goroutine 37 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc0012e8460}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc0012e8460}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0001aca80?}, {{0x824f9c8?, 0xc0004c0000?}, 0xc00063f340?}, 0xc00009b7e0, {0xc0014343c0, 0x15}, {0x7841a74, 0x14}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func23.2({0x7fbcf0684460, 0xc00066a2a0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:77
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc00066a2a0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 81
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83 @ 03/10/26 16:42:11.169
  "ts"="2026-03-10 16:42:11.169810" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.170007" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-5k975s" "resourceGroup"="np-autoscaling-cb9l84"
  "ts"="2026-03-10 16:42:12.605524" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-5k975s" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.934752" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-10 16:42:13.934928" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-10 16:42:13.935067" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.938167" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-146"
  "ts"="2026-03-10 16:42:13.938214" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-cb9l84"
  "ts"="2026-03-10 16:42:14.762169" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-146" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:15.414810" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-cb9l84" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
  "ts"="2026-03-10 16:42:16.145633" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-146" "deployment"="msi-deployment-146"
  "ts"="2026-03-10 16:42:16.860089" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-cb9l84" "deployment"="customer-infra-np-autoscale-cluster-5k975s"
  "ts"="2026-03-10 16:42:18.287841" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.288071" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.288315" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-cb9l84"
Command Error: signal: killed
Deserializaion Error: no output from command
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fb62b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-n5g8lq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000749e40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n5g8lq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000749e20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n5g8lq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000749e00>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n5g8lq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fb62b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-n5g8lq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000749e40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n5g8lq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000749e20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n5g8lq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000749e00>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n5g8lq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
13:59:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapErrors | 0xc00076e2a0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-rxmv5f, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rxmv5f" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-rxmv5f, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rxmv5f\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a16280>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-rxmv5f",
            },
            <*fmt.wrapError | 0xc000c56260>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rxmv5f\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapErrors | 0xc00076e2a0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-rxmv5f, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rxmv5f" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-rxmv5f, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rxmv5f\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a16280>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-rxmv5f",
            },
            <*fmt.wrapError | 0xc000c56260>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rxmv5f\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001580258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-qxsk8f" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000910a40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qxsk8f\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000910a20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qxsk8f\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009109e0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qxsk8f\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001580258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-qxsk8f" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000910a40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qxsk8f\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000910a20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qxsk8f\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009109e0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qxsk8f\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015802e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6lnzw5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0012ba200>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6lnzw5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0012ba1c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6lnzw5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0012ba1a0>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6lnzw5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015802e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6lnzw5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0012ba200>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6lnzw5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0012ba1c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6lnzw5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0012ba1a0>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6lnzw5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Deserializaion Error: no output from command
13:59:50 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Engineering should be able to retrieve kusto logs for a cluster and services (Spec Runtime: 1h29m59.619s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
        In [It] (Node Runtime: 1h29m59.619s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
          At [By Step] creating the HCP cluster (Step Runtime: 35m58.743s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:79

          Spec Goroutine
          goroutine 108 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000c484b0}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000c484b0}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0008903f0?}, {{0x824f9c8?, 0xc000a3ec00?}, 0xc001690258?}, 0xc00009b808, {0xc000082900, 0x11}, {0x7850b87, 0x17}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func22.2({0x7f6280596a60, 0xc0009642d0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:80
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc0009642d0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 105
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87 @ 03/10/26 16:42:11.152
  "ts"="2026-03-10 16:42:11.152421" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.152597" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-engineering-hcp-cluster-9tns56" "resourceGroup"="kusto-logs-4rrctc"
  "ts"="2026-03-10 16:42:13.106436" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-engineering-hcp-cluster-9tns56" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.813634" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz"
  "ts"="2026-03-10 16:42:13.813756" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz"
  "ts"="2026-03-10 16:42:13.813850" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.815776" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-138"
  "ts"="2026-03-10 16:42:13.815811" "level"=0 "msg"="collecting deployments" "resourceGroup"="kusto-logs-4rrctc"
  "ts"="2026-03-10 16:42:14.673064" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-138" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:15.383665" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-4rrctc" "deployment"="pooledNonMsiScopedAssignments-5borhjiq7mwd4"
  "ts"="2026-03-10 16:42:16.051229" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-138" "deployment"="msi-deployment-138"
  "ts"="2026-03-10 16:42:16.728183" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-4rrctc" "deployment"="customer-infra-engineering-hcp-cluster-9tns56"
  "ts"="2026-03-10 16:42:18.078393" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.078613" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.078797" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="kusto-logs-4rrctc"
Command Error: signal: killed
Deserializaion Error: no output from command
specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Engineering should be able to retrieve kusto logs for a cluster and services (Spec Runtime: 1h29m59.619s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
        In [It] (Node Runtime: 1h29m59.619s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
          At [By Step] creating the HCP cluster (Step Runtime: 35m58.743s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:79

          Spec Goroutine
          goroutine 108 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000c484b0}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000c484b0}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0008903f0?}, {{0x824f9c8?, 0xc000a3ec00?}, 0xc001690258?}, 0xc00009b808, {0xc000082900, 0x11}, {0x7850b87, 0x17}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func22.2({0x7f6280596a60, 0xc0009642d0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:80
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc0009642d0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 105
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87 @ 03/10/26 16:42:11.152
  "ts"="2026-03-10 16:42:11.152421" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.152597" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-engineering-hcp-cluster-9tns56" "resourceGroup"="kusto-logs-4rrctc"
  "ts"="2026-03-10 16:42:13.106436" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-engineering-hcp-cluster-9tns56" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.813634" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz"
  "ts"="2026-03-10 16:42:13.813756" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-fb6d207289fe1114ce672abf5865312a466e24e7ab0cc85e59e941314773e2c8.yaml.gz"
  "ts"="2026-03-10 16:42:13.813850" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.815776" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-138"
  "ts"="2026-03-10 16:42:13.815811" "level"=0 "msg"="collecting deployments" "resourceGroup"="kusto-logs-4rrctc"
  "ts"="2026-03-10 16:42:14.673064" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-138" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:15.383665" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-4rrctc" "deployment"="pooledNonMsiScopedAssignments-5borhjiq7mwd4"
  "ts"="2026-03-10 16:42:16.051229" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-138" "deployment"="msi-deployment-138"
  "ts"="2026-03-10 16:42:16.728183" "level"=0 "msg"="collecting operations" "resourceGroup"="kusto-logs-4rrctc" "deployment"="customer-infra-engineering-hcp-cluster-9tns56"
  "ts"="2026-03-10 16:42:18.078393" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.078613" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.078797" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="kusto-logs-4rrctc"
Command Error: signal: killed
Deserializaion Error: no output from command
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000810d38>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-kkw9j2" in resourcegroup="admin-credential-lifecycle-test-j4vxjb" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00059cd20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-kkw9j2\" in resourcegroup=\"admin-credential-lifecycle-test-j4vxjb\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00059cd00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-kkw9j2\" in resourcegroup=\"admin-credential-lifecycle-test-j4vxjb\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00059cce0>{
                        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-kkw9j2\" in resourcegroup=\"admin-credential-lifecycle-test-j4vxjb\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000810d38>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-kkw9j2" in resourcegroup="admin-credential-lifecycle-test-j4vxjb" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00059cd20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-kkw9j2\" in resourcegroup=\"admin-credential-lifecycle-test-j4vxjb\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00059cd00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-kkw9j2\" in resourcegroup=\"admin-credential-lifecycle-test-j4vxjb\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00059cce0>{
                        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-kkw9j2\" in resourcegroup=\"admin-credential-lifecycle-test-j4vxjb\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001044240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cg4txq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009bf640>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cg4txq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009bf620>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cg4txq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009bf600>{
                        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cg4txq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001044240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cg4txq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009bf640>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cg4txq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009bf620>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cg4txq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009bf600>{
                        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-cg4txq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
13:59:50 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:120]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010beed0>: 
    failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-brbm4q, error: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-brbm4q" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-brbm4q, error: failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-brbm4q\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011da740>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-brbm4q",
            },
            <*fmt.wrapError | 0xc00051a800>{
                msg: "failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-brbm4q\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:120]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010beed0>: 
    failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-brbm4q, error: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-brbm4q" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-brbm4q, error: failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-brbm4q\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011da740>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-brbm4q",
            },
            <*fmt.wrapError | 0xc00051a800>{
                msg: "failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-brbm4q\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001304138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-hhc27d" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000cff140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-hhc27d\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000cff120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-hhc27d\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000cff100>{
                        msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-hhc27d\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001304138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-hhc27d" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000cff140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-hhc27d\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000cff120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-hhc27d\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000cff100>{
                        msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-hhc27d\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Deserializaion Error: no output from command
13:59:50 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
skip all remaining specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request (Spec Runtime: 1h29m59.547s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:120
        In [It] (Node Runtime: 1h29m59.547s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:120
          At [By Step] creating the HCP cluster (Step Runtime: 33m59.309s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:152

          Spec Goroutine
          goroutine 30 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000032730}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000032730}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc00039b810?}, {{0x824f9c8?, 0xc000736000?}, 0xc000979f90?}, 0xc00009b798, {0xc00150a0a8, 0x11}, {0x783722d, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func13.2.1({0x7f4a7442c3c8, 0xc0008f8390})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:153
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc0008f8390?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 85
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160 @ 03/10/26 16:42:11.159
  "ts"="2026-03-10 16:42:11.159834" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.160053" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-tags-cluster-tqzd7z" "resourceGroup"="patch-tags-gz26w9"
  "ts"="2026-03-10 16:42:12.625579" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-tags-cluster-tqzd7z" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.498474" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz"
  "ts"="2026-03-10 16:42:13.498612" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz"
  "ts"="2026-03-10 16:42:13.498707" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.500587" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-66"
  "ts"="2026-03-10 16:42:13.500631" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-tags-gz26w9"
  "ts"="2026-03-10 16:42:14.310344" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-66" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:14.987312" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-gz26w9" "deployment"="pooledNonMsiScopedAssignments-fklo75htpxzxc"
  "ts"="2026-03-10 16:42:15.710879" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-66" "deployment"="msi-deployment-66"
  "ts"="2026-03-10 16:42:16.375074" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-gz26w9" "deployment"="customer-infra-patch-tags-cluster-tqzd7z"
  "ts"="2026-03-10 16:42:17.814778" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:17.815089" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:17.815329" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-tags-gz26w9"
Command Error: signal: killed
Deserializaion Error: no output from command
skip all remaining specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request (Spec Runtime: 1h29m59.547s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:120
        In [It] (Node Runtime: 1h29m59.547s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:120
          At [By Step] creating the HCP cluster (Step Runtime: 33m59.309s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:152

          Spec Goroutine
          goroutine 30 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000032730}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000032730}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc00039b810?}, {{0x824f9c8?, 0xc000736000?}, 0xc000979f90?}, 0xc00009b798, {0xc00150a0a8, 0x11}, {0x783722d, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func13.2.1({0x7f4a7442c3c8, 0xc0008f8390})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:153
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc0008f8390?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 85
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160 @ 03/10/26 16:42:11.159
  "ts"="2026-03-10 16:42:11.159834" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.160053" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-tags-cluster-tqzd7z" "resourceGroup"="patch-tags-gz26w9"
  "ts"="2026-03-10 16:42:12.625579" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-tags-cluster-tqzd7z" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.498474" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz"
  "ts"="2026-03-10 16:42:13.498612" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz"
  "ts"="2026-03-10 16:42:13.498707" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.500587" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-66"
  "ts"="2026-03-10 16:42:13.500631" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-tags-gz26w9"
  "ts"="2026-03-10 16:42:14.310344" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-66" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:14.987312" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-gz26w9" "deployment"="pooledNonMsiScopedAssignments-fklo75htpxzxc"
  "ts"="2026-03-10 16:42:15.710879" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-66" "deployment"="msi-deployment-66"
  "ts"="2026-03-10 16:42:16.375074" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-gz26w9" "deployment"="customer-infra-patch-tags-cluster-tqzd7z"
  "ts"="2026-03-10 16:42:17.814778" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:17.815089" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:17.815329" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-tags-gz26w9"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
13:59:50 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
ip all remaining specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Customer should be able to create node pools with ARM64-based VMs (Spec Runtime: 1h29m59.701s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:35
        In [It] (Node Runtime: 1h29m59.701s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:35
          At [By Step] creating the HCP cluster (Step Runtime: 35m59.6s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:82

          Spec Goroutine
          goroutine 106 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000a34aa0}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000a34aa0}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0004cebd0?}, {{0x824f9c8?, 0xc000668980?}, 0xc0000833c8?}, 0xc000f957e8, {0xc000d56090, 0x17}, {0x7841948, 0x14}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func4.2({0x7feeb4458260, 0xc000ce6a20})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:83
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc000ce6a20?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 103
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90 @ 03/10/26 16:42:11.165
  "ts"="2026-03-10 16:42:11.165938" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.166147" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-arm64-vm-hcp-cluster-dvlvjr" "resourceGroup"="arm64-vm-cluster-zqzxbn"
  "ts"="2026-03-10 16:42:13.109545" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-arm64-vm-hcp-cluster-dvlvjr" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.675589" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz"
  "ts"="2026-03-10 16:42:13.675720" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz"
  "ts"="2026-03-10 16:42:13.675847" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.678043" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-99"
  "ts"="2026-03-10 16:42:13.678091" "level"=0 "msg"="collecting deployments" "resourceGroup"="arm64-vm-cluster-zqzxbn"
  "ts"="2026-03-10 16:42:14.543605" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-99" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:15.262106" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-zqzxbn" "deployment"="pooledNonMsiScopedAssignments-6htyoibevj7ye"
  "ts"="2026-03-10 16:42:16.006348" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-99" "deployment"="msi-deployment-99"
  "ts"="2026-03-10 16:42:16.694697" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-zqzxbn" "deployment"="customer-infra-arm64-vm-hcp-cluster-dvlvjr"
  "ts"="2026-03-10 16:42:18.161866" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.162091" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.162364" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="arm64-vm-cluster-zqzxbn"
Command Error: signal: killed
Deserializaion Error: no output from command
ip all remaining specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Customer should be able to create node pools with ARM64-based VMs (Spec Runtime: 1h29m59.701s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:35
        In [It] (Node Runtime: 1h29m59.701s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:35
          At [By Step] creating the HCP cluster (Step Runtime: 35m59.6s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:82

          Spec Goroutine
          goroutine 106 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000a34aa0}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000a34aa0}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0004cebd0?}, {{0x824f9c8?, 0xc000668980?}, 0xc0000833c8?}, 0xc000f957e8, {0xc000d56090, 0x17}, {0x7841948, 0x14}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func4.2({0x7feeb4458260, 0xc000ce6a20})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:83
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc000ce6a20?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 103
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90 @ 03/10/26 16:42:11.165
  "ts"="2026-03-10 16:42:11.165938" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.166147" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-arm64-vm-hcp-cluster-dvlvjr" "resourceGroup"="arm64-vm-cluster-zqzxbn"
  "ts"="2026-03-10 16:42:13.109545" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-arm64-vm-hcp-cluster-dvlvjr" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.675589" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz"
  "ts"="2026-03-10 16:42:13.675720" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a6eb9d68aa9cc22f7e78141cf691002056d2ca6a7d34a8fba7d067b26fef9e86.yaml.gz"
  "ts"="2026-03-10 16:42:13.675847" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.678043" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-99"
  "ts"="2026-03-10 16:42:13.678091" "level"=0 "msg"="collecting deployments" "resourceGroup"="arm64-vm-cluster-zqzxbn"
  "ts"="2026-03-10 16:42:14.543605" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-99" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:15.262106" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-zqzxbn" "deployment"="pooledNonMsiScopedAssignments-6htyoibevj7ye"
  "ts"="2026-03-10 16:42:16.006348" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-99" "deployment"="msi-deployment-99"
  "ts"="2026-03-10 16:42:16.694697" "level"=0 "msg"="collecting operations" "resourceGroup"="arm64-vm-cluster-zqzxbn" "deployment"="customer-infra-arm64-vm-hcp-cluster-dvlvjr"
  "ts"="2026-03-10 16:42:18.161866" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.162091" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.162364" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="arm64-vm-cluster-zqzxbn"
Command Error: signal: killed
Deserializaion Error: no output from command
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012f81e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-f79m7k" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d18460>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-f79m7k\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d18440>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-f79m7k\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000fd94a0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-f79m7k\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012f81e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-f79m7k" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d18460>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-f79m7k\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d18440>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-f79m7k\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000fd94a0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-f79m7k\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded
13:59:50 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:294]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011def00>: 
    failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-sm9zjp" for cluster="cidr-connectivity-test" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sm9zjp, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"ext-auth-cidr\" in resourcegroup=\"e2e-cidr-connectivity-sm9zjp\" for cluster=\"cidr-connectivity-test\" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sm9zjp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00095cd20>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sm9zjp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:294]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011def00>: 
    failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-sm9zjp" for cluster="cidr-connectivity-test" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sm9zjp, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"ext-auth-cidr\" in resourcegroup=\"e2e-cidr-connectivity-sm9zjp\" for cluster=\"cidr-connectivity-test\" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sm9zjp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00095cd20>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sm9zjp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
13:59:50 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010506f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-tq274r" in resourcegroup="cluster-listing-zjw4z7" to finish deleting: context canceled
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-wdkbpb" in resourcegroup="cluster-listing-mm6jmt" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f87b80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-tq274r\" in resourcegroup=\"cluster-listing-zjw4z7\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f87b60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-tq274r\" in resourcegroup=\"cluster-listing-zjw4z7\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f87b40>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-tq274r\" in resourcegroup=\"cluster-listing-zjw4z7\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xc000f87c00>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-wdkbpb\" in resourcegroup=\"cluster-listing-mm6jmt\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f87be0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-wdkbpb\" in resourcegroup=\"cluster-listing-mm6jmt\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f87bc0>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-wdkbpb\" in resourcegroup=\"cluster-listing-mm6jmt\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010506f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-tq274r" in resourcegroup="cluster-listing-zjw4z7" to finish deleting: context canceled
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-wdkbpb" in resourcegroup="cluster-listing-mm6jmt" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f87b80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-tq274r\" in resourcegroup=\"cluster-listing-zjw4z7\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f87b60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-tq274r\" in resourcegroup=\"cluster-listing-zjw4z7\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f87b40>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-tq274r\" in resourcegroup=\"cluster-listing-zjw4z7\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xc000f87c00>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-wdkbpb\" in resourcegroup=\"cluster-listing-mm6jmt\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f87be0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-wdkbpb\" in resourcegroup=\"cluster-listing-mm6jmt\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f87bc0>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-wdkbpb\" in resourcegroup=\"cluster-listing-mm6jmt\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Deserializaion Error: no output from command
13:59:50 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
t again to skip cleanup.
    Here's a current progress report:
      Customer should be able to lifecycle and confirm external auth on a cluster (Spec Runtime: 1h29m59.553s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:43
        In [It] (Node Runtime: 1h29m59.553s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:43
          At [By Step] creating HCP cluster (Step Runtime: 34m58.072s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:77

          Spec Goroutine
          goroutine 31 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000b4a410}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000b4a410}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0006d6690?}, {{0x824f9c8?, 0xc000d70080?}, 0xc000f48550?}, 0xc00115b5e0, {0xc00022d300, 0xe}, {0xc000671470, 0xd}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func19.2({0x7f4c405d3288, 0xc000932300})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:78
                | 
                | By("creating HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc000932300?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 97
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84 @ 03/10/26 16:42:11.157
  "ts"="2026-03-10 16:42:11.158116" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.158267" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ea-listgbmwk6-qnr59m" "resourceGroup"="ea-list-8kkmkp"
  "ts"="2026-03-10 16:42:12.921428" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ea-listgbmwk6-qnr59m" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.881003" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz"
  "ts"="2026-03-10 16:42:13.881160" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz"
  "ts"="2026-03-10 16:42:13.881287" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.883374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-41"
  "ts"="2026-03-10 16:42:13.883427" "level"=0 "msg"="collecting deployments" "resourceGroup"="ea-list-8kkmkp"
  "ts"="2026-03-10 16:42:14.674750" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-8kkmkp" "deployment"="pooledNonMsiScopedAssignments-3wdubkoimyjyo"
  "ts"="2026-03-10 16:42:15.429771" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-41" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:16.199782" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-8kkmkp" "deployment"="customer-infra-ea-listgbmwk6-qnr59m"
  "ts"="2026-03-10 16:42:16.964086" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-41" "deployment"="msi-deployment-41"
  "ts"="2026-03-10 16:42:18.268034" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.268266" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.268491" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ea-list-8kkmkp"
Command Error: signal: killed
Deserializaion Error: no output from command
t again to skip cleanup.
    Here's a current progress report:
      Customer should be able to lifecycle and confirm external auth on a cluster (Spec Runtime: 1h29m59.553s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:43
        In [It] (Node Runtime: 1h29m59.553s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:43
          At [By Step] creating HCP cluster (Step Runtime: 34m58.072s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:77

          Spec Goroutine
          goroutine 31 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x823a1b8, 0xc000b4a410}, 0x781b40d?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x826d180, {0x823a1b8, 0xc000b4a410}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x823a228?, 0xc0006d6690?}, {{0x824f9c8?, 0xc000d70080?}, 0xc000f48550?}, 0xc00115b5e0, {0xc00022d300, 0xe}, {0xc000671470, 0xd}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77fbb54, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func19.2({0x7f4c405d3288, 0xc000932300})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:78
                | 
                | By("creating HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x824faa0?, 0xc000932300?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 97
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84 @ 03/10/26 16:42:11.157
  "ts"="2026-03-10 16:42:11.158116" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-10 16:42:11.158267" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ea-listgbmwk6-qnr59m" "resourceGroup"="ea-list-8kkmkp"
  "ts"="2026-03-10 16:42:12.921428" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ea-listgbmwk6-qnr59m" "resourceGroup"=""
  "ts"="2026-03-10 16:42:13.881003" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz"
  "ts"="2026-03-10 16:42:13.881160" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-399a7bed574b7c4dc2699f197c9d42dfc0866e60ee9f0a810da6aa8ecac44a02.yaml.gz"
  "ts"="2026-03-10 16:42:13.881287" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-10 16:42:13.883374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-41"
  "ts"="2026-03-10 16:42:13.883427" "level"=0 "msg"="collecting deployments" "resourceGroup"="ea-list-8kkmkp"
  "ts"="2026-03-10 16:42:14.674750" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-8kkmkp" "deployment"="pooledNonMsiScopedAssignments-3wdubkoimyjyo"
  "ts"="2026-03-10 16:42:15.429771" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-41" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-10 16:42:16.199782" "level"=0 "msg"="collecting operations" "resourceGroup"="ea-list-8kkmkp" "deployment"="customer-infra-ea-listgbmwk6-qnr59m"
  "ts"="2026-03-10 16:42:16.964086" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-41" "deployment"="msi-deployment-41"
  "ts"="2026-03-10 16:42:18.268034" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-10 16:42:18.268266" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-10 16:42:18.268491" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ea-list-8kkmkp"
Command Error: signal: killed
Deserializaion Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
13:59:50 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013c63c0>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-kdgvhl, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-kdgvhl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-kdgvhl, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kdgvhl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009b8460>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-kdgvhl",
            },
            <*fmt.wrapError | 0xc000a12240>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kdgvhl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013c63c0>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-kdgvhl, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-kdgvhl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-kdgvhl, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kdgvhl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009b8460>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-kdgvhl",
            },
            <*fmt.wrapError | 0xc000a12240>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kdgvhl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head e7f1e37 · merge e7f1e37
 n/a n/a passed #4244 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9653ebd
 n/a unknown failed #4305 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7c96f74
 n/a e2e failed #4198 (open) 16
e2e (16)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001644198>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listwz5kkc" in resourcegroup="ea-list-764klc" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000b751c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listwz5kkc\" in resourcegroup=\"ea-list-764klc\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000b751a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listwz5kkc\" in resourcegroup=\"ea-list-764klc\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000b75180>{
                        msg: "failed waiting for hcpCluster=\"ea-listwz5kkc\" in resourcegroup=\"ea-list-764klc\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001644198>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listwz5kkc" in resourcegroup="ea-list-764klc" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000b751c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listwz5kkc\" in resourcegroup=\"ea-list-764klc\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000b751a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listwz5kkc\" in resourcegroup=\"ea-list-764klc\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000b75180>{
                        msg: "failed waiting for hcpCluster=\"ea-listwz5kkc\" in resourcegroup=\"ea-list-764klc\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:47:47 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f7ee70>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-q9wg6x" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-q9wg6x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00153ea90>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x",
            },
            <*fmt.wrapErrors | 0xc000f7ee40>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-q9wg6x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00153ea90>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f7ee70>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-q9wg6x" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-q9wg6x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00153ea90>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x",
            },
            <*fmt.wrapErrors | 0xc000f7ee40>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-q9wg6x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00153ea90>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-q9wg6x",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f1d638>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m7xjc2" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0005257c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7xjc2\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0005257a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7xjc2\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000525780>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7xjc2\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f1d638>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m7xjc2" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0005257c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7xjc2\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0005257a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7xjc2\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000525780>{
                        msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7xjc2\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:47:47 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:242]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012ef050>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-k2bh7f, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k2bh7f" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-k2bh7f, error: failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-k2bh7f\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00074d310>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-k2bh7f",
            },
            <*fmt.wrapError | 0xc0009d8780>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-k2bh7f\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:242]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012ef050>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-k2bh7f, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k2bh7f" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-k2bh7f, error: failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-k2bh7f\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00074d310>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-k2bh7f",
            },
            <*fmt.wrapError | 0xc0009d8780>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-k2bh7f\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Cluster should become ready within 45 minutes
12:47:47 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0008984f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-dd8bvh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000990540>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-dd8bvh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000990520>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-dd8bvh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000990500>{
                        msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-dd8bvh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0008984f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-dd8bvh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000990540>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-dd8bvh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000990520>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-dd8bvh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000990500>{
                        msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-dd8bvh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000d82720>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-grbcml" in resourcegroup="cluster-listing-8lw594" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0016413c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-grbcml\" in resourcegroup=\"cluster-listing-8lw594\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0016413a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-grbcml\" in resourcegroup=\"cluster-listing-8lw594\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001641380>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-grbcml\" in resourcegroup=\"cluster-listing-8lw594\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000d82720>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-grbcml" in resourcegroup="cluster-listing-8lw594" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0016413c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-grbcml\" in resourcegroup=\"cluster-listing-8lw594\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0016413a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-grbcml\" in resourcegroup=\"cluster-listing-8lw594\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001641380>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-grbcml\" in resourcegroup=\"cluster-listing-8lw594\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00170e360>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-g8cnxk" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000745c80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-g8cnxk\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000745c40>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-g8cnxk\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000745c20>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-g8cnxk\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00170e360>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-g8cnxk" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000745c80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-g8cnxk\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000745c40>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-g8cnxk\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000745c20>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-g8cnxk\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:47:47 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:129]: Unexpected error:
    <*fmt.wrapErrors | 0xc00148e450>: 
    failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-sxs7cw, error: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-sxs7cw" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-sxs7cw, error: failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-sxs7cw\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001371a80>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-sxs7cw",
            },
            <*fmt.wrapError | 0xc0007602e0>{
                msg: "failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-sxs7cw\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:129]: Unexpected error:
    <*fmt.wrapErrors | 0xc00148e450>: 
    failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-sxs7cw, error: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-sxs7cw" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-one-node, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-sxs7cw, error: failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-sxs7cw\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001371a80>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-one-node in resource group nodepool-update-nodes-sxs7cw",
            },
            <*fmt.wrapError | 0xc0007602e0>{
                msg: "failed waiting for nodepool=\"np-one-node\" for cluster \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-sxs7cw\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0016821b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-k8q9jw" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007658a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-k8q9jw\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000765880>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-k8q9jw\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000765860>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-k8q9jw\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0016821b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-k8q9jw" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007658a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-k8q9jw\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000765880>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-k8q9jw\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000765860>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-k8q9jw\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:47:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011e8fc0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lbmfqb, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-lbmfqb" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lbmfqb, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lbmfqb\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00103a230>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lbmfqb",
            },
            <*fmt.wrapError | 0xc000a12e20>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lbmfqb\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011e8fc0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lbmfqb, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-lbmfqb" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lbmfqb, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lbmfqb\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00103a230>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-lbmfqb",
            },
            <*fmt.wrapError | 0xc000a12e20>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-lbmfqb\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000bf2240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5lmvwj" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000ea9260>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-5lmvwj\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000ea9240>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-5lmvwj\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000ea9220>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-5lmvwj\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000bf2240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5lmvwj" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000ea9260>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-5lmvwj\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000ea9240>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-5lmvwj\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000ea9220>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-5lmvwj\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:47:47 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00122b050>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-kg59h8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-kg59h8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e79440>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8",
            },
            <*fmt.wrapErrors | 0xc00122b020>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-kg59h8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e79440>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00122b050>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-kg59h8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-kg59h8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e79440>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8",
            },
            <*fmt.wrapErrors | 0xc00122b020>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-kg59h8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e79440>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-kg59h8",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:47:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015201f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-pb2tlz" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000409400>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-pb2tlz\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0004093e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-pb2tlz\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0004093c0>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-pb2tlz\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015201f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-pb2tlz" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000409400>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-pb2tlz\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0004093e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-pb2tlz\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0004093c0>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-pb2tlz\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb0640c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:47:47 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc001084720>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-6rrcd4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-6rrcd4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001711760>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4",
            },
            <*fmt.wrapErrors | 0xc0010846f0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-6rrcd4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001711760>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc001084720>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-6rrcd4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-6rrcd4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001711760>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4",
            },
            <*fmt.wrapErrors | 0xc0010846f0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-6rrcd4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001711760>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-6rrcd4",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:47:47 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fd5650>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-bjwrl4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bjwrl4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ee9b60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4",
            },
            <*fmt.wrapErrors | 0xc000fd5620>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bjwrl4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ee9b60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fd5650>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-bjwrl4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bjwrl4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ee9b60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4",
            },
            <*fmt.wrapErrors | 0xc000fd5620>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-bjwrl4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ee9b60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-bjwrl4",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5d66b41 · merge 5d66b41
 n/a n/a passed #4387 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0398ab0 · merge 0398ab0
 n/a n/a passed #4386 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f76ac24 · merge 76945bf
 n/a n/a passed #4384 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 7a2f8f6 · merge 1892906
 n/a e2e failed #3820 (merged) 2
e2e (2)
ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift
11:53:30 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc000858d00>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-c7v86b" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54",
      "name": "73d75999-13f8-466e-b285-0185af102f54",
      "status": "Failed",
      "startTime": "2026-03-10T12:52:29.933777459Z",
      "endTime": "2026-03-10T12:54:22.810279491Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:730cd492-d01e-007b-1e8c-b0ca06000000\nTime:2026-03-10T12:53:58.1157027Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c7v86b\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\",\n  \"name\": \"73d75999-13f8-466e-b285-0185af102f54\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T12:52:29.933777459Z\",\n  \"endTime\": \"2026-03-10T12:54:22.810279491Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:730cd492-d01e-007b-1e8c-b0ca06000000\\nTime:2026-03-10T12:53:58.1157027Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000858ca0>{
            msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c7v86b\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\",\n  \"name\": \"73d75999-13f8-466e-b285-0185af102f54\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T12:52:29.933777459Z\",\n  \"endTime\": \"2026-03-10T12:54:22.810279491Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:730cd492-d01e-007b-1e8c-b0ca06000000\\nTime:2026-03-10T12:53:58.1157027Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc000858d00>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-c7v86b" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54",
      "name": "73d75999-13f8-466e-b285-0185af102f54",
      "status": "Failed",
      "startTime": "2026-03-10T12:52:29.933777459Z",
      "endTime": "2026-03-10T12:54:22.810279491Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:730cd492-d01e-007b-1e8c-b0ca06000000\nTime:2026-03-10T12:53:58.1157027Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c7v86b\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\",\n  \"name\": \"73d75999-13f8-466e-b285-0185af102f54\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T12:52:29.933777459Z\",\n  \"endTime\": \"2026-03-10T12:54:22.810279491Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:730cd492-d01e-007b-1e8c-b0ca06000000\\nTime:2026-03-10T12:53:58.1157027Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000858ca0>{
            msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c7v86b\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d75999-13f8-466e-b285-0185af102f54\",\n  \"name\": \"73d75999-13f8-466e-b285-0185af102f54\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T12:52:29.933777459Z\",\n  \"endTime\": \"2026-03-10T12:54:22.810279491Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:730cd492-d01e-007b-1e8c-b0ca06000000\\nTime:2026-03-10T12:53:58.1157027Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift
11:53:30 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapError | 0xc000da4b40>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g7c879" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd",
      "name": "0cf03c0e-11b2-4fad-9131-9411f43c03cd",
      "status": "Failed",
      "startTime": "2026-03-10T13:09:21.351421482Z",
      "endTime": "2026-03-10T13:10:35.544610896Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:f23c12f7-c01e-00f9-558f-b09298000000\nTime:2026-03-10T13:10:32.3633238Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-g7c879\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"name\": \"0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T13:09:21.351421482Z\",\n  \"endTime\": \"2026-03-10T13:10:35.544610896Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f23c12f7-c01e-00f9-558f-b09298000000\\nTime:2026-03-10T13:10:32.3633238Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000da4b20>{
            msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-g7c879\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"name\": \"0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T13:09:21.351421482Z\",\n  \"endTime\": \"2026-03-10T13:10:35.544610896Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f23c12f7-c01e-00f9-558f-b09298000000\\nTime:2026-03-10T13:10:32.3633238Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
       ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapError | 0xc000da4b40>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g7c879" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd",
      "name": "0cf03c0e-11b2-4fad-9131-9411f43c03cd",
      "status": "Failed",
      "startTime": "2026-03-10T13:09:21.351421482Z",
      "endTime": "2026-03-10T13:10:35.544610896Z",
      "error": {
        "code": "InternalServerError",
        "message": "GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:f23c12f7-c01e-00f9-558f-b09298000000\nTime:2026-03-10T13:10:32.3633238Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-g7c879\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"name\": \"0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T13:09:21.351421482Z\",\n  \"endTime\": \"2026-03-10T13:10:35.544610896Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f23c12f7-c01e-00f9-558f-b09298000000\\nTime:2026-03-10T13:10:32.3633238Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000da4b20>{
            msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-g7c879\" to finish creating: GET https://rp.j5696256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"name\": \"0cf03c0e-11b2-4fad-9131-9411f43c03cd\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-10T13:09:21.351421482Z\",\n  \"endTime\": \"2026-03-10T13:10:35.544610896Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"GET https://arohcpprowoidcj5696256.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:f23c12f7-c01e-00f9-558f-b09298000000\\nTime:2026-03-10T13:10:32.3633238Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
       ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 69b2b21 · merge 69b2b21
 n/a n/a passed #4380 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head e680d6c · merge 506b932
 n/a n/a passed #4372 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head e680d6c · merge 506b932
 n/a e2e failed ★#4372 (merged) 1
e2e (1)
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error
11:26:30 UTC · failed at e2e · test Engineering should be able to retrieve expected metrics from the /metrics endpoint · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc000988880>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc0008e67e0>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc000988880>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc0008e67e0>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 40bc11d · merge 40bc11d
 n/a n/a passed #4313 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 40bc11d · merge 40bc11d
 n/a n/a passed #4313 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 59be665 · merge 76945bf
 n/a unknown failed #4384 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5ce5146
 n/a provision failed ⚠#4381 (closed) 1
provision (1)
error running Image Mirror Step, failed to fetch pull secret: no credentials found for registry registry.k8s.io in pull secret
10:50:01 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/global/mirror-image · suite step graph
Show raw failure
time=2026-03-10T11:41:32.333Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T11:41:32.667Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-10T11:41:33.068Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to fetch pull secret: no credentials found for registry registry.k8s.io in pull secret"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head eb269d3 · merge 9128c3e
 n/a e2e failed ⚠#4383 (merged) 1
e2e (1)
VerifyHostedControlPlaneVersionUpgradeFrom(initial=4
10:21:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade using the candidate channel · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:111]: Timed out after 2400.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f882b8>: 
    VerifyHostedControlPlaneVersionUpgradeFrom(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
    {
        errs: [
            <*fmt.wrapError | 0xc0009544e0>{
                msg: "VerifyHostedControlPlaneVersionUpgradeFrom(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                err: <*errors.errorString | 0xc0014725e0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:111]: Timed out after 2400.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f882b8>: 
    VerifyHostedControlPlaneVersionUpgradeFrom(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
    {
        errs: [
            <*fmt.wrapError | 0xc0009544e0>{
                msg: "VerifyHostedControlPlaneVersionUpgradeFrom(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                err: <*errors.errorString | 0xc0014725e0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                },
            },
        ],
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head e680d6c · merge 506b932
 n/a provision failed ⚠#4372 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code DeploymentScriptOperationFailed; detail message A task was canceled.
10:20:39 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph
Show raw failure
time=2026-03-10T10:55:50.130Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-10T10:55:53.522Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-10T10:55:55.220Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=ea9c323e7e0ed5e03e9ed7dc554923094a25757910fec39e816eea2c3c50a0e9
time=2026-03-10T11:08:58.308Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5557376-svc/providers/Microsoft.Resources/deployments/ea9c323e7e0ed5e03e9ed7dc554923094a25757910fec39e816eea2c3c50a0e9/operationStatuses/08584284667311545815\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/sessiongate-cert-fvino4gogvxv2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deploymentScripts/newCert-sessiongate-cert-prow-j5557376\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentScriptOperationFailed\\\",\\r\\n                \\\"message\\\": \\\"A task was canceled.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 59be665 · merge 76945bf
 n/a unknown failed #4384 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 11ecb87 · merge e0b528c
 n/a e2e failed #4371 (merged) 3
e2e (3)
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift
10:15:57 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: Unexpected error:
    <*fmt.wrapError | 0xc000354780>: 
    failed to create NodePool autoscale-np: failed starting nodepool creation "autoscale-np" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rqk2f4": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool autoscale-np: failed starting nodepool creation \"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rqk2f4\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000354740>{
            msg: "failed starting nodepool creation \"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rqk2f4\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00039ede0>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["94"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "8a8ac7c7-a51b-4490-9f41-b6b26b9ba143",
                        ],
                        "Date": [
                            "Tue, 10 Mar 2026 11:19:17 GMT",
                        ],
                        "Content-Length": ["345"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000354580>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: Unexpected error:
    <*fmt.wrapError | 0xc000354780>: 
    failed to create NodePool autoscale-np: failed starting nodepool creation "autoscale-np" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rqk2f4": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool autoscale-np: failed starting nodepool creation \"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rqk2f4\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000354740>{
            msg: "failed starting nodepool creation \"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rqk2f4\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-rqk2f4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-np\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00039ede0>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["94"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "8a8ac7c7-a51b-4490-9f41-b6b26b9ba143",
                        ],
                        "Date": [
                            "Tue, 10 Mar 2026 11:19:17 GMT",
                        ],
                        "Content-Length": ["345"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000354580>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: Conflict; detail message Cannot create resource while parent resource is provisioning; provider Microsoft.RedHatOpenShift
10:15:57 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapError | 0xc00116a620>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-cnkhnk": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-cnkhnk\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00116a600>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-cnkhnk\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000a93320>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Date": [
                            "Tue, 10 Mar 2026 11:19:25 GMT",
                        ],
                        "Content-Length": ["340"],
                        "X-Envoy-Upstream-Service-Time": ["2114"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "0a972fe0-75c5-45bf-a114-5bca2d0dace1",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc00116a520>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapError | 0xc00116a620>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-cnkhnk": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot create resource while parent resource is provisioning",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-cnkhnk\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00116a600>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-cnkhnk\": PUT https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot create resource while parent resource is provisioning\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cnkhnk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000a93320>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Date": [
                            "Tue, 10 Mar 2026 11:19:25 GMT",
                        ],
                        "Content-Length": ["340"],
                        "X-Envoy-Upstream-Service-Time": ["2114"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "0a972fe0-75c5-45bf-a114-5bca2d0dace1",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc00116a520>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 99, 114, 101, 97, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 119, 104, 105, 108, 101, 32, 112, 97, 114, 101, 110, 116, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: Conflict; detail message Cannot request credential while resource is "provisioning"; provider Microsoft.RedHatOpenShift
10:15:57 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:92]: Unexpected error:
    <*fmt.wrapError | 0xc000778540>: 
    failed to start credential request: POST https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0007a5170>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "3ac49bae-07be-4982-9e71-34184dd26057",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 11:19:25 GMT",
                    ],
                    "Content-Length": ["336"],
                    "X-Envoy-Upstream-Service-Time": ["2038"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000778440>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 110, 111, 100, 101, 112, 111, 111, 108, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 98, 120, 104, 118, 112, 115, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 110, 112, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 336,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j9932288.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/974ebd46-8ad3-41e3-afef...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:92]: Unexpected error:
    <*fmt.wrapError | 0xc000778540>: 
    failed to start credential request: POST https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://rp.j9932288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while resource is \\\"provisioning\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-bxhvps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0007a5170>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "X-Ms-Request-Id": [
                        "3ac49bae-07be-4982-9e71-34184dd26057",
                    ],
                    "Date": [
                        "Tue, 10 Mar 2026 11:19:25 GMT",
                    ],
                    "Content-Length": ["336"],
                    "X-Envoy-Upstream-Service-Time": ["2038"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000778440>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 32, 105, 115, 32, 92, 34, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 92, 34, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 110, 111, 100, 101, 112, 111, 111, 108, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 98, 120, 104, 118, 112, 115, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 104, 99, 112, 79, 112, 101, 110, 83, 104, 105, 102, 116, 67, 108, 117, 115, 116, 101, 114, 115, 47, 110, 112, 45, 117, 112, 100, 97, 116, 101, 45, 110, 111, 100, 101, 115, 45, 104, 99, 112, 45, 99, 108, 117, 115, 116, 101, 114, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 336,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j9932288.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/974ebd46-8ad3-41e3-afef...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0289e5c · merge 0289e5c
 n/a n/a passed #4329 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 88b6e6b · merge 69b2b21
 n/a provision failed ⚠#4380 (merged) 1
provision (1)
resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress context deadline exceeded
09:53:37 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/aro-hcp-exporter · suite step graph
Show raw failure
time=2026-03-10T10:40:57.288Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter description="Step aro-hcp-exporter\n Kind: Helm\n"
time=2026-03-10T10:40:57.770Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter values="map[cacheTTL:60m collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:[service-tag-usage] image:map[digest:sha256:2a64b8900486cd886d2c44023788f95891a3012ad790f85272432a30685e2a34 registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:9df125b8-466a-45c8-a7b7-9eeb6183562f name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-10T10:40:57.770Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T10:40:58.100Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter namespace=aro-hcp-exporter
time=2026-03-10T10:40:58.340Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter dryRun=false
time=2026-03-10T10:40:58.340Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T10:40:58.456Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T10:46:01.741Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-10T10:46:01.741Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T10:46:01.741Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T10:46:04.076Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter release=aro-hcp-exporter namespace=aro-hcp-exporter status=failed description="Release \"aro-hcp-exporter\" failed: resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[cacheTTL:60m collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:[service-tag-usage] image:map[digest:sha256:2a64b8900486cd886d2c44023788f95891a3012ad790f85272432a30685e2a34 registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:9df125b8-466a-45c8-a7b7-9eeb6183562f name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-10T10:46:04.078Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yRQWsqMRSF9/kVl1k5MCOj7z15THFRaFel3bQrSxeZ5KDRmdyQXLWCP75MpdUWhNrtl3tOkvupFkIRidfRINGUrBYtumkx4GYJI3fO2zpJdH5e0AE96A6fCBv4d5KCNh84f1ZERNkj4sYZ3LN3wjErKNORy4UJJV4DR8EZ9iV9bQyvvfwyfWnsBqHlXYcLLlQvV2q1bhA9BOm2X0hSe9ouEEHiOiTRXaAGsgU8DawW9HiQjavxpKz+lKPqaVTVf6v63/9ZltNwSGdnJnU1mmV5rva0ZOdp5bydOu8RTzSyp6O9U23ffak9hcj96fGhBQW2h+GflRTU8rygDinpeV/J0SJSszv5vEUybwEAAP//A4U572wCAAA="
time=2026-03-10T10:46:04.080Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T10:46:04.080Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head e7f1e37 · merge e7f1e37
 n/a n/a passed #4244 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 523a271 · merge 523a271
 n/a n/a passed #4363 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 18b2acc · merge 18b2acc
 n/a n/a passed #4379 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head c7c9d81 · merge c7c9d81
 n/a n/a passed #4366 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0f5cde2 · merge 0f5cde2
 n/a n/a passed #4382 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head a77910e
 n/a provision failed ⚠#4381 (closed) 1
provision (1)
error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform "FetchReference" on source: GET "<url>": response status code 401: unauthorized: authentication required: [map[Action:pull Class
08:16:57 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/global/mirror-image · suite step graph
Show raw failure
time=2026-03-10T09:01:37.170Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T09:01:37.468Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-10T09:01:38.017Z level=INFO msg="Failed to fetch pull secret, will try anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image error="no credentials found for registry registry.build02.ci.openshift.org in pull secret"
time=2026-03-10T09:01:38.017Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-10T09:01:39.590Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image source=registry.build02.ci.openshift.org/ci-op-9bchqrkl/pipeline@sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f target=arohcpsvcdev.azurecr.io/ci-op-9bchqrkl/pipeline:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f
time=2026-03-10T09:01:39.697Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:02:09.773Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:02:39.861Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:03:39.862Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T09:03:40.158Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-10T09:03:40.768Z level=INFO msg="Failed to fetch pull secret, will try anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image error="no credentials found for registry registry.build02.ci.openshift.org in pull secret"
time=2026-03-10T09:03:40.768Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-10T09:03:42.242Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image source=registry.build02.ci.openshift.org/ci-op-9bchqrkl/pipeline@sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f target=arohcpsvcdev.azurecr.io/ci-op-9bchqrkl/pipeline:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f
time=2026-03-10T09:03:42.293Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:04:12.489Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:04:42.539Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:05:42.540Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T09:05:42.830Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-10T09:05:43.385Z level=INFO msg="Failed to fetch pull secret, will try anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image error="no credentials found for registry registry.build02.ci.openshift.org in pull secret"
time=2026-03-10T09:05:43.385Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-10T09:05:45.031Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image source=registry.build02.ci.openshift.org/ci-op-9bchqrkl/pipeline@sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f target=arohcpsvcdev.azurecr.io/ci-op-9bchqrkl/pipeline:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f
time=2026-03-10T09:05:45.053Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:06:15.089Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:06:45.142Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:07:45.143Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T09:07:45.513Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-10T09:07:46.191Z level=INFO msg="Failed to fetch pull secret, will try anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image error="no credentials found for registry registry.build02.ci.openshift.org in pull secret"
time=2026-03-10T09:07:46.191Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-10T09:07:47.689Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image source=registry.build02.ci.openshift.org/ci-op-9bchqrkl/pipeline@sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f target=arohcpsvcdev.azurecr.io/ci-op-9bchqrkl/pipeline:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f
time=2026-03-10T09:07:47.777Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:08:17.828Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:08:47.892Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:09:47.894Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-10T09:09:48.182Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-10T09:09:48.931Z level=INFO msg="Failed to fetch pull secret, will try anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image error="no credentials found for registry registry.build02.ci.openshift.org in pull secret"
time=2026-03-10T09:09:48.931Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-10T09:09:50.464Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image source=registry.build02.ci.openshift.org/ci-op-9bchqrkl/pipeline@sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f target=arohcpsvcdev.azurecr.io/ci-op-9bchqrkl/pipeline:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f
time=2026-03-10T09:09:50.531Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:10:20.600Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:10:50.718Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
time=2026-03-10T09:11:50.723Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"FetchReference\" on source: GET \"https://registry.build02.ci.openshift.org/v2/ci-op-9bchqrkl/pipeline/manifests/sha256:c982aa0bfa01bbbecab3f1f77e10f1aae554ededa3d0d30fada49049423a091f\": response status code 401: unauthorized: authentication required: [map[Action:pull Class: Name:ci-op-9bchqrkl/pipeline Type:repository]]"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 88b6e6b · merge 69b2b21
 n/a provision failed ⚠#4380 (merged) 1
provision (1)
resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress context deadline exceeded
08:09:15 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/aro-hcp-exporter · suite step graph
Show raw failure
time=2026-03-10T09:14:23.191Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter description="Step aro-hcp-exporter\n Kind: Helm\n"
time=2026-03-10T09:14:23.917Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter values="map[cacheTTL:60m collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:[service-tag-usage] image:map[digest:sha256:2a64b8900486cd886d2c44023788f95891a3012ad790f85272432a30685e2a34 registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:e577ae7c-f860-47b0-91ba-c263f6adcacc name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-10T09:14:23.917Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T09:14:24.102Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter namespace=aro-hcp-exporter
time=2026-03-10T09:14:24.322Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter dryRun=false
time=2026-03-10T09:14:24.322Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T09:14:24.429Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T09:19:27.307Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-10T09:19:27.308Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T09:19:27.308Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T09:19:29.663Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter release=aro-hcp-exporter namespace=aro-hcp-exporter status=failed description="Release \"aro-hcp-exporter\" failed: resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[cacheTTL:60m collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:[service-tag-usage] image:map[digest:sha256:2a64b8900486cd886d2c44023788f95891a3012ad790f85272432a30685e2a34 registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:e577ae7c-f860-47b0-91ba-c263f6adcacc name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-03-10T09:19:29.664Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yRQU/yQBCG7/0Vk55o0pLSj3yGGg4mejJ60RPGw3b3DSy0O5vdASThx5tKFGIkEq/Pvu/s7jxJC6GAyOugEWlKRokS1bQYcLOElnvrTB0lWDfP6YAeVYcvhA3cB4le6U+cvSREROktfMu7Dk7SnFIVuFhoX+DNcxCEn9mh+YSwsRoP7KzwmeSv7Rutef3nuy+uJa/XyWrdIDgI4l2/kJjsabtAAIntEEV1nhrIFnA0MErQ40FaldX/ovxXjMrnclKPxnU1nqUZDYd0NjOpq6tZmmXJnpZsHa2sM1PrHMKJRnZ0tHeq7buvZE8+cH96fGhOns0hfNmQnFqe59QhRjXvR3IwCNTsTj5vEPV7AAAA//+dHcuZbAIAAA=="
time=2026-03-10T09:19:29.668Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter
time=2026-03-10T09:19:29.669Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-exporter, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 40bc11d · merge 40bc11d
 n/a n/a passed #4313 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 523a271 · merge 523a271
 n/a n/a passed #4363 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c7c9d81 · merge c7c9d81
 n/a n/a passed #4366 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 18b2acc · merge 18b2acc
 n/a n/a passed #4379 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 86cf6ca · merge 86cf6ca
 n/a n/a passed #4378 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 66a5158
 n/a n/a passed #4272 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c7c9d81 · merge c7c9d81
 n/a n/a passed #4366 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head e680d6c · merge 506b932
 n/a n/a passed #4372 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 86cf6ca · merge 86cf6ca
 n/a n/a passed #4378 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head d1097e4 · merge d1097e4
 n/a n/a passed #4374 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee08914 · merge ee08914
 n/a n/a passed #4377 (merged) 0 n/a