CIHealth
<
Last 7 Days▾
Last 7 Days Weekly: Oct 12 - Oct 18 Sprint: Oct 12 - Oct 25 Single Day: Oct 13
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 111 · Failures matched: 875 · Total runs: 311 · Runs affected: 220

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail code KeyVaultAccessForbidden; detail message Unable to access key vault resource '<url>' to enable encryption at rest. Please grant get, wrap ...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail code KeyVaultAccessForbidden; detail message Unable to access key vault resource '<url>' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x14dd1e091060>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-kdz84x79jt45" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "name": "75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "status": "Failed",
      "startTime": "2026-10-12T12:59:06.888844222Z",
      "endTime": "2026-10-12T13:28:08.456404461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine des-encrypt-des-np-f8895-79vxf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine des-encrypt-des-np-f8895-gfvrv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine des-encrypt-des-np-f8895-79vxf: Failed: virtualmachine failed to create or update. err: failed to create or update resource des-encrypt-kdz84x79jt45--managed/des-encrypt-des-np-f8895-79vxf (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-kdz84x79jt45--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-f8895-79vxf\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: KeyVaultAccessForbidden\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"KeyVaultAccessForbidden\",\n    \"message\": \"Unable to access key vault resource 'https://cust-kv-knxkedtao2ry2.vault.azure.net/keys/des-encryption-key/3126b361c5834ce5ad1e97723eb63855' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
            correlation ID: bd8a92be-4962-42e7-8e34-639e73d20045
            client request ID: 6eda585f-7541-486b-8dad-d5cd134191ff
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x14dd1e091060>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-kdz84x79jt45" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "name": "75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "status": "Failed",
      "startTime": "2026-10-12T12:59:06.888844222Z",
      "endTime": "2026-10-12T13:28:08.456404461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine des-encrypt-des-np-f8895-79vxf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine des-encrypt-des-np-f8895-gfvrv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine des-encrypt-des-np-f8895-79vxf: Failed: virtualmachine failed to create or update. err: failed to create or update resource des-encrypt-kdz84x79jt45--managed/des-encrypt-des-np-f8895-79vxf (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-kdz84x79jt45--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-f8895-79vxf\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: KeyVaultAccessForbidden\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"KeyVaultAccessForbidden\",\n    \"message\": \"Unable to access key vault resource 'https://cust-kv-knxkedtao2ry2.vault.azure.net/keys/des-encryption-key/3126b361c5834ce5ad1e97723eb63855' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
            correlation ID: bd8a92be-4962-42e7-8e34-639e73d20045
            client request ID: 6eda585f-7541-486b-8dad-d5cd134191ff
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74702109611047785598976
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mvkgs4" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2777a0f273c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mvkgs4: failed waiting for cluster="cp-ystream-upgrade-4-22-mvkgs4" in resourcegroup="rg-cp-ystream-upgrade-4-22-blssk6qzfvws" to finish creating: GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "name": "a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:53.687204284Z",
      "endTime": "2026-10-12T16:21:54.576185121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-policy-controller, kube-controller-manager, openshift-apiserver, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, certified-operators-catalog, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
            correlation ID: f4045377-6462-4996-b798-a9fd2a096fae
            client request ID: 01fbf192-7565-417e-a2b4-3d30deb4b951
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mvkgs4" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2777a0f273c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mvkgs4: failed waiting for cluster="cp-ystream-upgrade-4-22-mvkgs4" in resourcegroup="rg-cp-ystream-upgrade-4-22-blssk6qzfvws" to finish creating: GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "name": "a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:53.687204284Z",
      "endTime": "2026-10-12T16:21:54.576185121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-policy-controller, kube-controller-manager, openshift-apiserver, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, certified-operators-catalog, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
            correlation ID: f4045377-6462-4996-b798-a9fd2a096fae
            client request ID: 01fbf192-7565-417e-a2b4-3d30deb4b951
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73832109662114334380032
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Un...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x495f5a50100>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-vtqpftbtl8pd" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "name": "b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.565546726Z",
      "endTime": "2026-10-12T16:34:19.299056423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
            correlation ID: 2f0e8af3-4477-4dd1-b4a2-0bb21a3ecc71
            client request ID: af43345c-178c-4f70-985a-42472759437b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x495f5a50100>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-vtqpftbtl8pd" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "name": "b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.565546726Z",
      "endTime": "2026-10-12T16:34:19.299056423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
            correlation ID: 2f0e8af3-4477-4dd1-b4a2-0bb21a3ecc71
            client request ID: af43345c-178c-4f70-985a-42472759437b
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0xe13d9ba2880>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-9czhr6hdgkds" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "name": "abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:22.752877991Z",
      "endTime": "2026-10-12T19:14:30.175613398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
            correlation ID: e99f3924-5824-4d14-bfb3-a5c2c635c6fb
            client request ID: 3e3fb9a9-7a4c-4e3b-9ae6-64fae2a5931a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0xe13d9ba2880>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-9czhr6hdgkds" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "name": "abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:22.752877991Z",
      "endTime": "2026-10-12T19:14:30.175613398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
            correlation ID: e99f3924-5824-4d14-bfb3-a5c2c635c6fb
            client request ID: 3e3fb9a9-7a4c-4e3b-9ae6-64fae2a5931a
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74672109704412896169984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '<timestamp>' with correlationId '<uuid>', and will expire at '<timestamp>' if it does not complete before then.
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:88]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xd4464611460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-b4xvfj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"DeploymentActive\",\r\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2hklr4tmfmv6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/12/2026 6:44:17 PM' with correlationId '483786d1-9a79-473c-9660-1216b08e7f90', and will expire at '10/19/2026 6:44:17 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
            correlation ID: d9670e5b-9049-4df2-b486-a7bed21f23b8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:88]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xd4464611460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-b4xvfj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"DeploymentActive\",\r\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2hklr4tmfmv6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/12/2026 6:44:17 PM' with correlationId '483786d1-9a79-473c-9660-1216b08e7f90', and will expire at '10/19/2026 6:44:17 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
            correlation ID: d9670e5b-9049-4df2-b486-a7bed21f23b8
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74672109704412896169984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDB
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
time=2026-10-12T23:56:30.391Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-12T23:56:32.064Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-12T23:56:32.955Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=2f9fa65a0ba704015b11015a72cc4e4b4675fce22bd699e4c355e7650345c6e3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3444352%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2f9fa65a0ba704015b11015a72cc4e4b4675fce22bd699e4c355e7650345c6e3
time=2026-10-13T00:03:04.471Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3444352/providers/Microsoft.Resources/deployments/2f9fa65a0ba704015b11015a72cc4e4b4675fce22bd699e4c355e7650345c6e3/operationStatuses/08584097574930206657\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3444352/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3444352/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j3444352\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n                \\\"message\\\": \\\"Database account creation failed. Operation Id: 42664daf-4e8e-4916-9d75-23b7169468da, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 26eaccdb-de17-4cbf-afc1-0c865cfbb1a2, Request URI: /apps/de70b74f-b005-4de6-b2c3-b39046bf9084/services/74353989-92b9-4a77-a49a-b47600be8b14/partitions/a40f5539-0cc6-43c1-b5bc-fdb8dcb67485/replicas/134362021132042649s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-10-13T00:02:20.9415661Z, RequestEndTime: 2026-10-13T00:02:26.9109610Z,  Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-13T00:01:28.0336227Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.273,\\\\\\\"memory\\\\\\\":21877192.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1501,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":48},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-13T00:01:38.0493247Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.521,\\\\\\\"memory\\\\\\\":21840180.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0747,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":48},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-13T00:01:48.0505987Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.332,\\\\\\\"memory\\\\\\\":21844420.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1332,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":48},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-13T00:01:58.0664654Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.975,\\\\\\\"memory\\\\\\\":21830384.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1746,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":48},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-13T00:02:08.0821422Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.332,\\\\\\\"memory\\\\\\\":21833552.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0846,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":48},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-13T00:02:18.0977872Z\\\\\\\",\\\\\\\"cpu\\\\\\\":2.281,\\\\\\\"memory\\\\\\\":21805124.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1832,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":51}]}\\\\r\\\\nRequestStart: 2026-10-13T00:02:20.9415661Z; ResponseTime: 2026-10-13T00:02:26.9109610Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-centralus1-be424.documents.azure.com:11306/apps/de70b74f-b005-4de6-b2c3-b39046bf9084/services/74353989-92b9-4a77-a49a-b47600be8b14/partitions/a40f5539-0cc6-43c1-b5bc-fdb8dcb67485/replicas/134362021132042649s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5975.386, ActivityId: 26eaccdb-de17-4cbf-afc1-0c865cfbb1a2, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11306 | status: Connected | lkt: 10/13/2026 12:02:26 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-13T00:02:20.9415661Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0013},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-13T00:02:20.9415674Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.006},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-13T00:02:20.9415734Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0628},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-13T00:02:20.9416362Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5976.0734},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-13T00:02:26.9177096Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1891},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-13T00:02:26.9178987Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-10-13T00:00:07.6413599Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-10-13T00:00:07.6413599Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-10-13T00:00:12.9085149Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":867,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74832109790061913444352
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
full failure pattern:
VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xb019e0de2e8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xb019c86fe00>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c86fda0>{
                    msg: "failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df811d0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xb019c4d6ff0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019bfab1e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019bfab0e0>{
                    msg: "failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df140c0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xb019c5e2a50>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019c4e0980>{
                msg: "VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c4e0960>{
                    msg: "failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019e2caff0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default",
                        Err: <*net.OpError | 0xb019bfc47d0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xb019e0de2e8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xb019c86fe00>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c86fda0>{
                    msg: "failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df811d0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xb019c4d6ff0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019bfab1e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019bfab0e0>{
                    msg: "failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df140c0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xb019c5e2a50>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019c4e0980>{
                msg: "VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c4e0960>{
                    msg: "failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019e2caff0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default",
                        Err: <*net.OpError | 0xb019bfc47d0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71622109675831700754432
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired
full failure pattern:
alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.32%1 of 311 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:24:31Z
  Ended:       2026-10-12T16:44:33Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption High - arohcpci01-rp-j9626368 does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert6219.94%62 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsOct 6: 15 · Oct 7: 29 · Oct 8: 19 · Oct 9: 4 · Oct 10: 0 · Oct 11: 2 · Oct 12: 8 · Oct 13: 0none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T20:59:27Z
  Ended:       2026-10-12T21:19:23Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6043392-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:22:14Z
  Ended:       2026-10-12T16:44:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9626368-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T14:00:36Z
  Ended:       2026-10-12T14:17:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4606592-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T14:00:36Z
  Ended:       2026-10-12T14:17:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4606592-svc", component="backend", name="csstatedump", region="centralus", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire62
Affected runs (62)
Date (UTC)Associated PRProw job
PR #74722109729919306043392
PR #74772109664686109626368
PR #71622109629712874606592
PR #67462109622919087788032
PR #74672109612109212946432
PR #74702109611047785598976
PR #74682109600459105767424
PR #67462109555123024826368
PR #74572109331063628107776
PR #74572109291724151984128
PR #74392108648873588166656
PR #74442108569711095582720
PR #73972108452968792592384
PR #74372108374328641851392
PR #73892108317197410504704
PR #71802108317092523544576
PR #74142108256251258343424
PR #73892108256213476052992
PR #72322108229382379147264
PR #73892108165661279653888
PR #74012108163022403932160
PR #73832108153534082977792
PR #73642108152878592954368
PR #63642108152449343688704
PR #72882108130674367008768
PR #73872108128762271895552
PR #74202108123255498870784
PR #74142108110245078241280
PR #74142108055963964542976
PR #71772108043334072143872
PR #74122108031501605015552
PR #73252108018145892503552
PR #72322108015628659986432
PR #74172107967978476670976
PR #74142107964840596213760
PR #63642107926490833227776
PR #73052107926353130033152
PR #74142107916465603612672
PR #74122107914072107257856
PR #72882107904235558408192
PR #72482107883389313880064
PR #68072107880778506440704
PR #71432107875990943305728
PR #73222107875761300967424
PR #74002107875420446658560
PR #73102107872947002675200
PR #74102107844183950626816
PR #74042107834229550223360
PR #63642107828301232869376
PR #74052107802095104036864
PR #66882107778619966230528
PR #71332107752436901875712
PR #74032107751899905134592
PR #73972107751278686769152
PR #71432107750854802018304
PR #73992107750792218808320
PR #73712107750686056779776
PR #73702107750168211230720
PR #69682107750153682161664
PR #73102107739528360890368
PR #73972107681070840287232
PR #70972107680144293367808
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
e2e3711.90%37 of 311 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 6: 236 · Oct 7: 126 · Oct 8: 12 · Oct 9: 9 · Oct 10: 2 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x23a798e4e3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7kr9hzf26ffd" to finish creating: GET https://rp.j0742656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357",
      "name": "2a131d64-b48f-453b-b60d-f4063709a357",
      "status": "Failed",
      "startTime": "2026-10-10T03:18:06.524337179Z",
      "endTime": "2026-10-10T03:47:07.577893958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x23a798e4e3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7kr9hzf26ffd" to finish creating: GET https://rp.j0742656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357",
      "name": "2a131d64-b48f-453b-b60d-f4063709a357",
      "status": "Failed",
      "startTime": "2026-10-10T03:18:06.524337179Z",
      "endTime": "2026-10-10T03:47:07.577893958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x560689fa080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-jzlxwvqzg694" to finish creating: GET https://rp.j5842176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "name": "efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "status": "Failed",
      "startTime": "2026-10-10T02:25:01.899620825Z",
      "endTime": "2026-10-10T02:54:06.245842706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x560689fa080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-jzlxwvqzg694" to finish creating: GET https://rp.j5842176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "name": "efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "status": "Failed",
      "startTime": "2026-10-10T02:25:01.899620825Z",
      "endTime": "2026-10-10T02:54:06.245842706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (29)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions15
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion13
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.2213
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.113
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.2112
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.012
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API12
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.2011
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public10
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.239
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with a private key vault using an available API version1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a fully private cluster with private KAS and private ingress and verify both are only reachable from VNet1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a fully private cluster with private KAS, private ingress, and private KeyVault and verify all are operational1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (37)
Date (UTC)Associated PRProw job
PR #74392108739904140742656
PR #71772108729709905842176
PR #74272108618645423788032
PR #74012108538835301306368
PR #73972108513285430054912
PR #69442108480044224483328
PR #73972108452968792592384
PR #74382108402844888469504
PR #73642108377905737240576
PR #63642108259306926247936
PR #73872108256216495951872
PR #74312108220296300335104
PR #73612108209233152446464
PR #73892108165661279653888
PR #63642108152449343688704
PR #71582108129603208876032
PR #72922108122723405271040
PR #73892108109354774302720
PR #74122108092712069632000
PR #71772108043334072143872
PR #69542107939651057618944
PR #73252107916955439599616
PR #72482107883389313880064
PR #68072107880778506440704
PR #74142107879778638893056
PR #71432107875990943305728
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #66882107778619966230528
PR #74052107765914286952448
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
PR #73102107739528360890368
PR #73972107681070840287232
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
alert247.72%24 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsOct 6: 6 · Oct 7: 9 · Oct 8: 3 · Oct 9: 2 · Oct 10: 1 · Oct 11: 4 · Oct 12: 5 · Oct 13: 0none
Full failure examples (3)
alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T20:51:04Z
  Ended:       2026-10-12T20:57:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6043392-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdfeve1hnh3ip3571ig9kpg8ogs5h0t-r4w0d6n8e5u2v7j", pod="etcd-2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tdfeve1hnh3ip3571ig9kpg8ogs5h0t-r4w0d6n8e5u2v7j/etcd-2 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:16:01Z
  Ended:       2026-10-12T16:22:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0290944-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdbebgjal25ks2eu7kru50siu2d9fk7-d5m6i4f8v4g8r0f", pod="kube-controller-manager-655fcf6674-rv7x7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tdbebgjal25ks2eu7kru50siu2d9fk7-d5m6i4f8v4g8r0f/kube-controller-manager-655fcf6674-rv7x7 has been in a non-ready state for longer than 5 minutes.
alert fired 20 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:30:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o", pod="azure-disk-csi-driver-controller-67b988dbd6-9pjgh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o/azure-disk-csi-driver-controller-67b988dbd6-9pjgh has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:25:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d", pod="multus-admission-controller-86cf4f5c69-mqbpc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d/multus-admission-controller-86cf4f5c69-mqbpc has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:25:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d", pod="multus-admission-controller-86cf4f5c69-jhk97", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d/multus-admission-controller-86cf4f5c69-jhk97 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o", pod="azure-disk-csi-driver-controller-67b988dbd6-b9n46", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o/azure-disk-csi-driver-controller-67b988dbd6-b9n46 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o", pod="azure-disk-csi-driver-controller-67b988dbd6-cg97m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o/azure-disk-csi-driver-controller-67b988dbd6-cg97m has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:25:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d", pod="multus-admission-controller-86cf4f5c69-kpf7b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d/multus-admission-controller-86cf4f5c69-kpf7b has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-12T13:13:45Z
  Ended:       2026-10-12T13:24:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i", pod="openshift-apiserver-6b4ff7ff95-9kn96", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i/openshift-apiserver-6b4ff7ff95-9kn96 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-7cvh9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-7cvh9 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-8mskw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-8mskw has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-j42hz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-j42hz has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-mkdz4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-mkdz4 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-5xwvh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-5xwvh has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-npkzv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-npkzv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-m8dx4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-m8dx4 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-pnr6f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-pnr6f has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-gw6tj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-gw6tj has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-5x6tc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-5x6tc has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-6lcvg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-6lcvg has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:24:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i", pod="openshift-apiserver-6b4ff7ff95-mksfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i/openshift-apiserver-6b4ff7ff95-mksfs has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-bx6hr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-bx6hr has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire24
Affected runs (24)
Date (UTC)Associated PRProw job
PR #74722109729919306043392
PR #73732109660256270290944
PR #74722109611255084879872
PR #74702109611047785598976
PR #74302109519598771507200
PR #74392109426221556174848
PR #74242109420238498959360
PR #74572109370651696435200
PR #64522109140137769177088
PR #73742108819747389837312
PR #74392108694594228064256
PR #72482108564560276361216
PR #73872108256216495951872
PR #72322108229382379147264
PR #74122108031501605015552
PR #63642107926490833227776
PR #73992107908786537631744
PR #72482107883389313880064
PR #74112107879778164936704
PR #74102107844183950626816
PR #63642107828301232869376
PR #73972107751278686769152
PR #74022107717187903950848
PR #74002107676117312737280
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert227.07%22 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 6: 7 · Oct 7: 2 · Oct 8: 7 · Oct 9: 6 · Oct 10: 0 · Oct 11: 0 · Oct 12: 7 · Oct 13: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:07:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3618048-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/hcps-j3618048", namespace="ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nlt8k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:07:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3618048-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/hcps-j3618048", namespace="ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nlt8k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:06:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2519552-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2519552/providers/microsoft.monitor/accounts/hcps-j2519552", namespace="ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nbzvl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2519552-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2519552/providers/microsoft.monitor/accounts/hcps-j2519552", namespace="ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nbzvl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:44:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:44:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:45:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:45:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire22
Affected runs (22)
Date (UTC)Associated PRProw job
PR #74572109743222543618048
PR #69202109740763322519552
PR #74722109611255084879872
PR #74702109611047785598976
PR #74682109600459105767424
PR #74632109563550279143424
PR #74242109528755037802496
PR #74482108692920105177088
PR #69042108611968167317504
PR #74402108607415241936896
PR #74372108513309102706688
PR #72922108438042090934272
PR #71052108409414854643712
PR #72922108317188833153024
PR #71802108317092523544576
PR #63642108259306926247936
PR #73892108256213476052992
PR #74222108197422965460992
PR #73832108153534082977792
PR #74142108055963964542976
PR #74142107879778638893056
PR #71432107875990943305728
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
e2e216.75%21 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsOct 6: 0 · Oct 7: 15 · Oct 8: 1 · Oct 9: 1 · Oct 10: 1 · Oct 11: 1 · Oct 12: 3 · Oct 13: 0PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xaa70fc27a80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-pcvlks6fdkqz" to finish creating: GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "name": "26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:23.241617081Z",
      "endTime": "2026-10-12T16:21:32.102166958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-q4ttl-4vcvt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-pcvlks6fdkqz--managed/ingress-cert-le-np-1-q4ttl-4vcvt-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-q4ttl-4vcvt-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: RetryableError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"RetryableError\",\n    \"message\": \"A retryable error occurred.\",\n    \"details\": [\n      {\n        \"code\": \"RetryableErrorDueToTooManyCalls\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX was used to perform too many calls within last 5 minutes. The number of ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
            correlation ID: bfc832f2-b15d-4a34-8c01-edecd304fc1a
            client request ID: cd0488d8-da8e-467d-8248-7320c5456b91
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xaa70fc27a80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-pcvlks6fdkqz" to finish creating: GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "name": "26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:23.241617081Z",
      "endTime": "2026-10-12T16:21:32.102166958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-q4ttl-4vcvt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-pcvlks6fdkqz--managed/ingress-cert-le-np-1-q4ttl-4vcvt-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-q4ttl-4vcvt-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: RetryableError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"RetryableError\",\n    \"message\": \"A retryable error occurred.\",\n    \"details\": [\n      {\n        \"code\": \"RetryableErrorDueToTooManyCalls\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX was used to perform too many calls within last 5 minutes. The number of ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
            correlation ID: bfc832f2-b15d-4a34-8c01-edecd304fc1a
            client request ID: cd0488d8-da8e-467d-8248-7320c5456b91
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:160]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2462d4d4a5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5hsxwfczjwp5" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "name": "466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "status": "Failed",
      "startTime": "2026-10-12T07:57:35.126724285Z",
      "endTime": "2026-10-12T08:26:42.188209535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
            correlation ID: df9f6d3b-b689-4825-a253-151f9584e7e4
            client request ID: 5fd0c7fe-22bf-4239-aeb7-b1d20c9c430c
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:160]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2462d4d4a5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5hsxwfczjwp5" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "name": "466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "status": "Failed",
      "startTime": "2026-10-12T07:57:35.126724285Z",
      "endTime": "2026-10-12T08:26:42.188209535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
            correlation ID: df9f6d3b-b689-4825-a253-151f9584e7e4
            client request ID: 5fd0c7fe-22bf-4239-aeb7-b1d20c9c430c
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc252182d240>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c847fqslv6ss" to finish creating: GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1",
      "name": "de9482bc-af9d-45d5-a419-3811860ecab1",
      "status": "Failed",
      "startTime": "2026-10-12T03:33:37.063419631Z",
      "endTime": "2026-10-12T04:02:41.894551349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
            correlation ID: 5048e7cb-ed4b-40b9-bfb8-b714b05f3bec
            client request ID: a436b4d6-2bc5-40f5-a8b1-84635292bdaa
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc252182d240>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c847fqslv6ss" to finish creating: GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1",
      "name": "de9482bc-af9d-45d5-a419-3811860ecab1",
      "status": "Failed",
      "startTime": "2026-10-12T03:33:37.063419631Z",
      "endTime": "2026-10-12T04:02:41.894551349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
            correlation ID: 5048e7cb-ed4b-40b9-bfb8-b714b05f3bec
            client request ID: a436b4d6-2bc5-40f5-a8b1-84635292bdaa
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool18
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.11
Affected runs (21)
Date (UTC)Associated PRProw job
PR #73112109657273612636160
PR #74402109536551896616960
PR #74392109474078921330688
PR #72192109252833235701760
PR #71652108710710912487424
PR #72922108377922371850240
PR #74222108197422965460992
PR #73252107916955439599616
PR #72482107883389313880064
PR #68072107880778506440704
PR #74142107879778638893056
PR #71432107875990943305728
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #74052107765914286952448
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
PR #73972107681070840287232
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
full failure pattern:
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e216.75%21 of 311 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsOct 6: 6 · Oct 7: 5 · Oct 8: 7 · Oct 9: 4 · Oct 10: 0 · Oct 11: 0 · Oct 12: 5 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-xqmd9dg8sc2s/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-xqmd9dg8sc2s/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4cmzlj277gjf/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4cmzlj277gjf/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-mb49f2vf2mck/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-mb49f2vf2mck/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation21
Affected runs (21)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
PR #69202109658742915076096
PR #74722109611255084879872
PR #73732109603048882966528
PR #74572109599905499582464
PR #72482108661352737280000
PR #74442108611234440941568
PR #74402108607415241936896
PR #74372108452755680006144
PR #74142108317119643914240
PR #72322108291662634553344
PR #71802108256210653286400
PR #71052108207385897078784
PR #73892108109354774302720
PR #67752108104508235059200
PR #74142108055963964542976
PR #74152107928469479362560
PR #72882107904235558408192
PR #72272107836958540566528
PR #73982107769248779603968
PR #73102107739528360890368
alert [svc] LeaderElectionLeaseStale fired
full failure pattern:
alert [svc] LeaderElectionLeaseStale fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert185.79%18 of 311 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsOct 6: 6 · Oct 7: 3 · Oct 8: 6 · Oct 9: 2 · Oct 10: 1 · Oct 11: 1 · Oct 12: 5 · Oct 13: 0none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T23:02:04Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j6774400-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.46:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6774400/providers/microsoft.monitor/accounts/services-j6774400", namespace="klusterlet-2tdgk5uq966q724hns0udujljsi1q0qu", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nxprp", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tdgk5uq966q724hns0udujljsi1q0qu on cluster ci01-j6774400-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:24:20Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3618048-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.226:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/services-j3618048", namespace="klusterlet-2tdg2drvs0bj4f4vu4h2o5drume4j1rc", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x9h2r", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tdg2drvs0bj4f4vu4h2o5drume4j1rc on cluster ci01-j3618048-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:44:16Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1015808-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.213:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1015808/providers/microsoft.monitor/accounts/services-j1015808", namespace="klusterlet-2td5olqiam9j0qubto977v59hbae6bo7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-m2h4p", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2td5olqiam9j0qubto977v59hbae6bo7 on cluster ci01-j1015808-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire18
Affected runs (18)
Date (UTC)Associated PRProw job
PR #73732109754180636774400
PR #74572109743222543618048
PR #72192109566058561015808
PR #74582109528983438626816
PR #74242109528755037802496
PR #71622109391747573354496
PR #73742108886756110635008
PR #72272108694589719187456
PR #74462108673315018117120
PR #73612108256282895978496
PR #72322108229382379147264
PR #73892108165661279653888
PR #73642108152878592954368
PR #73782108101483491758080
PR #74122108092712069632000
PR #73612107979674851217408
PR #74122107955248256520192
PR #74142107916465603612672
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e144.50%14 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsOct 6: 39 · Oct 7: 16 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2de8dabda108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v5jj9lf2wr67" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "name": "c17139f2-d108-4953-bf0d-a162f95b9ed1",
      "status": "Failed",
      "startTime": "2026-10-07T18:20:36.247617787Z",
      "endTime": "2026-10-07T18:49:41.673001167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zwd2g-2w92c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-2w92c to exist\nMachine cilium-cluster-cilium-np-zwd2g-xq8t6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v5jj9lf2wr67--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-zwd2g-xq8t6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features9
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully7
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR2
Affected runs (14)
Date (UTC)Associated PRProw job
PR #72922108377922371850240
PR #74372108374328641851392
PR #72482107883389313880064
PR #68072107880778506440704
PR #71432107875990943305728
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #74052107765914286952448
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
PR #73972107681070840287232
alert [svc] KustoLogsDataStale fired
full failure pattern:
alert [svc] KustoLogsDataStale fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
alert123.86%12 of 311 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsOct 6: 5 · Oct 7: 4 · Oct 8: 8 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:22:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:23:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire12
Affected runs (12)
Date (UTC)Associated PRProw job
PR #73892108317197410504704
PR #73642108317114874990592
PR #74142108256251258343424
PR #73712108248385097043968
PR #71622108221120938250240
PR #73872108181128597737472
PR #63642108152449343688704
PR #72882108130674367008768
PR #72882107979694195347456
PR #68072107880778506440704
PR #74072107818144989974528
PR #74002107676117312737280
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e92.89%9 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsOct 6: 0 · Oct 7: 3 · Oct 8: 4 · Oct 9: 6 · Oct 10: 0 · Oct 11: 0 · Oct 12: 3 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3aa92878a060>: 
    resource group "hcp-metrics-w7n8h467qbnb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-w7n8h467qbnb" to finish deleting: GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103",
      "name": "eb261c81-ca83-45b2-bdfd-0de40227c103",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:20.608938335Z",
      "endTime": "2026-10-12T16:22:21.07258074Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5jcbgo4cpr0su6el4fc6i0lag0dj still exists (deletion dispatched at 2026-10-12T15:58:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
            correlation ID: 9da4d8e2-ebfd-4094-bf43-c7701d90768a
            client request ID: 61ad067a-2c1b-43a6-b798-69bd268d4471
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3aa92878a060>: 
    resource group "hcp-metrics-w7n8h467qbnb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-w7n8h467qbnb" to finish deleting: GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103",
      "name": "eb261c81-ca83-45b2-bdfd-0de40227c103",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:20.608938335Z",
      "endTime": "2026-10-12T16:22:21.07258074Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5jcbgo4cpr0su6el4fc6i0lag0dj still exists (deletion dispatched at 2026-10-12T15:58:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
            correlation ID: 9da4d8e2-ebfd-4094-bf43-c7701d90768a
            client request ID: 61ad067a-2c1b-43a6-b798-69bd268d4471
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x1476512fa078>: 
    resource group "rg-cluster-nsg-subnet-reuse-xjg677xdgtps": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xjg677xdgtps" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "name": "d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:47.33491695Z",
      "endTime": "2026-10-12T16:22:51.509357112Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb58jqqdu8adgelm46n5haqq88idqc still exists (deletion dispatched at 2026-10-12T15:58:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
            correlation ID: f7d4d296-e895-4942-ace3-b59e68657889
            client request ID: 314b1f02-1204-4002-a941-604f43078239
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x1476512fa078>: 
    resource group "rg-cluster-nsg-subnet-reuse-xjg677xdgtps": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xjg677xdgtps" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "name": "d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:47.33491695Z",
      "endTime": "2026-10-12T16:22:51.509357112Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb58jqqdu8adgelm46n5haqq88idqc still exists (deletion dispatched at 2026-10-12T15:58:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
            correlation ID: f7d4d296-e895-4942-ace3-b59e68657889
            client request ID: 314b1f02-1204-4002-a941-604f43078239
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x351b8e4ae138>: 
    resource group "disabled-image-registry-pzs7vs68g9xt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-pzs7vs68g9xt" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13",
      "name": "f1e860d3-9723-4468-b91f-a96474515d13",
      "status": "Failed",
      "startTime": "2026-10-12T15:57:28.987630883Z",
      "endTime": "2026-10-12T16:21:31.7469599Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5a4g859cjm0hvqmru36k5u916p18 still exists (deletion dispatched at 2026-10-12T15:57:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
            correlation ID: eab286f4-49cc-48ea-8ec4-6c70b43ff1d0
            client request ID: 1a74a385-2e63-4deb-a488-b16ad26cc354
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x351b8e4ae138>: 
    resource group "disabled-image-registry-pzs7vs68g9xt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-pzs7vs68g9xt" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13",
      "name": "f1e860d3-9723-4468-b91f-a96474515d13",
      "status": "Failed",
      "startTime": "2026-10-12T15:57:28.987630883Z",
      "endTime": "2026-10-12T16:21:31.7469599Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5a4g859cjm0hvqmru36k5u916p18 still exists (deletion dispatched at 2026-10-12T15:57:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
            correlation ID: eab286f4-49cc-48ea-8ec4-6c70b43ff1d0
            client request ID: 1a74a385-2e63-4deb-a488-b16ad26cc354
occurred
Contributing tests (15)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (9)
Date (UTC)Associated PRProw job
PR #74512109659008674566144
PR #69202109658742915076096
PR #74412108512930990395392
PR #71622108511709156085760
PR #74142108377968924430336
PR #69282108311064448864256
PR #73612108209233152446464
PR #74142108110245078241280
PR #74002107875420446658560
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert92.89%9 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 6: 5 · Oct 7: 1 · Oct 8: 2 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 4 · Oct 13: 0none
Full failure examples (3)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T21:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3618048-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/hcps-j3618048", namespace="ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o", pod="cloud-network-config-controller-54f58dd464-7xc6g", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d482c703-1325-4911-a179-7fd115deb112"
  Description: Pod ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o/cloud-network-config-controller-54f58dd464-7xc6g (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T21:56:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3618048-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/hcps-j3618048", namespace="ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o", pod="cloud-network-config-controller-54f58dd464-7xc6g", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d482c703-1325-4911-a179-7fd115deb112"
  Description: Pod ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o/cloud-network-config-controller-54f58dd464-7xc6g (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T21:57:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3618048-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/hcps-j3618048", namespace="ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o", pod="csi-snapshot-controller-746f546586-q8jss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8db2a082-8106-487f-b148-74ed6ac93307"
  Description: Pod ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o/csi-snapshot-controller-746f546586-q8jss (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-12T21:59:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3618048-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3618048/providers/microsoft.monitor/accounts/hcps-j3618048", namespace="ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o", pod="csi-snapshot-controller-746f546586-q8jss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8db2a082-8106-487f-b148-74ed6ac93307"
  Description: Pod ocm-arohcpci01-2tdg2k4p3a0tbs6v291jvkejb9gp79a4-s8e1t1g8l8t9e0o/csi-snapshot-controller-746f546586-q8jss (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:45:53Z
  Ended:       2026-10-12T20:13:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9ddee187-d906-4221-b48f-73e1102c7b1e"
  Description: Pod ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-12T19:45:53Z
  Ended:       2026-10-12T20:13:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9ddee187-d906-4221-b48f-73e1102c7b1e"
  Description: Pod ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:08:11Z
  Ended:       2026-10-12T13:34:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4d015a5d-c4df-4754-bdd2-c1b853805f1f"
  Description: Pod ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:09:11Z
  Ended:       2026-10-12T13:33:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4d015a5d-c4df-4754-bdd2-c1b853805f1f"
  Description: Pod ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire9
Affected runs (9)
Date (UTC)Associated PRProw job
PR #74572109743222543618048
PR #74792109715328622661632
PR #74702109611047785598976
PR #74632109563550279143424
PR #74482108692920105177088
PR #72922108438042090934272
PR #71802108317092523544576
PR #74142108055963964542976
PR #74142107879778638893056
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e82.57%8 of 311 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 1 · Oct 9: 8 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x643438c6138>: 
    resource group "des-encrypt-mfwpss6hqvsd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="des-encrypt" in resourcegroup="des-encrypt-mfwpss6hqvsd" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234",
      "name": "a58b36b6-30fb-4aee-839d-12ae7b447234",
      "status": "Failed",
      "startTime": "2026-10-12T16:03:15.661006285Z",
      "endTime": "2026-10-12T16:27:21.295309747Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
            correlation ID: 0827ef63-2c1b-4cad-8799-bd4b41067d42
            client request ID: 2c486539-fc98-4112-bf4c-2bdff7d36160
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x643438c6138>: 
    resource group "des-encrypt-mfwpss6hqvsd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="des-encrypt" in resourcegroup="des-encrypt-mfwpss6hqvsd" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234",
      "name": "a58b36b6-30fb-4aee-839d-12ae7b447234",
      "status": "Failed",
      "startTime": "2026-10-12T16:03:15.661006285Z",
      "endTime": "2026-10-12T16:27:21.295309747Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
            correlation ID: 0827ef63-2c1b-4cad-8799-bd4b41067d42
            client request ID: 2c486539-fc98-4112-bf4c-2bdff7d36160
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x87a272e80c0>: 
    resource group "autoscaling-cluster-dch49t4bvhmd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dch49t4bvhmd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "name": "314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:02.340187908Z",
      "endTime": "2026-10-09T12:35:03.752595501Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x87a272e80c0>: 
    resource group "autoscaling-cluster-dch49t4bvhmd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dch49t4bvhmd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "name": "314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:02.340187908Z",
      "endTime": "2026-10-09T12:35:03.752595501Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f620761c090>: 
    resource group "customer-rg-lbqh9vzm8skr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lbqh9vzm8skr" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253",
      "name": "ff9dce59-52f3-4407-ad07-f8872d204253",
      "status": "Failed",
      "startTime": "2026-10-09T12:06:09.323431375Z",
      "endTime": "2026-10-09T12:30:13.733939626Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f620761c090>: 
    resource group "customer-rg-lbqh9vzm8skr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lbqh9vzm8skr" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253",
      "name": "ff9dce59-52f3-4407-ad07-f8872d204253",
      "status": "Failed",
      "startTime": "2026-10-09T12:06:09.323431375Z",
      "endTime": "2026-10-09T12:30:13.733939626Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
Affected runs (8)
Date (UTC)Associated PRProw job
PR #69202109658742915076096
PR #74372108513309102706688
PR #74412108512930990395392
PR #71622108511709156085760
PR #74142108377968924430336
PR #74372108374328641851392
PR #71052108207385897078784
PR #71432107875990943305728
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e61.93%6 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsOct 6: 0 · Oct 7: 3 · Oct 8: 1 · Oct 9: 5 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x715418a8198>: 
    resource group "rg-hs-pre--d8fntzv77rg5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6jnw6j" in resourcegroup="rg-hs-pre--d8fntzv77rg5" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa",
      "name": "1f6364b4-168b-4408-93de-ce5f903b20aa",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:02.733087351Z",
      "endTime": "2026-10-09T12:29:06.620380667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kg3h59n4o9tks40jdsh5l5jro8g still exists (deletion dispatched at 2026-10-09T12:05:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x715418a8198>: 
    resource group "rg-hs-pre--d8fntzv77rg5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6jnw6j" in resourcegroup="rg-hs-pre--d8fntzv77rg5" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa",
      "name": "1f6364b4-168b-4408-93de-ce5f903b20aa",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:02.733087351Z",
      "endTime": "2026-10-09T12:29:06.620380667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kg3h59n4o9tks40jdsh5l5jro8g still exists (deletion dispatched at 2026-10-09T12:05:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x52f9b8920c0>: 
    resource group "cni-cilium-lqwdfr6cjvht": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-lqwdfr6cjvht" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "name": "d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "status": "Failed",
      "startTime": "2026-10-09T11:57:36.843017136Z",
      "endTime": "2026-10-09T12:21:46.074977182Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qasjq198a6vq4pknlkm4b3grgda still exists (deletion dispatched at 2026-10-09T11:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x52f9b8920c0>: 
    resource group "cni-cilium-lqwdfr6cjvht": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-lqwdfr6cjvht" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "name": "d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "status": "Failed",
      "startTime": "2026-10-09T11:57:36.843017136Z",
      "endTime": "2026-10-09T12:21:46.074977182Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qasjq198a6vq4pknlkm4b3grgda still exists (deletion dispatched at 2026-10-09T11:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x4a16d2b40f0>: 
    resource group "rg-candidate-5-1-trdpp8hx8tpd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jjf49v" in resourcegroup="rg-candidate-5-1-trdpp8hx8tpd" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "name": "292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:28.202107106Z",
      "endTime": "2026-10-09T03:29:29.34914283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh1i1h74jbtnnmtcppkg9ambqd8h still exists (deletion dispatched at 2026-10-09T03:05:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x4a16d2b40f0>: 
    resource group "rg-candidate-5-1-trdpp8hx8tpd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jjf49v" in resourcegroup="rg-candidate-5-1-trdpp8hx8tpd" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "name": "292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:28.202107106Z",
      "endTime": "2026-10-09T03:29:29.34914283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh1i1h74jbtnnmtcppkg9ambqd8h still exists (deletion dispatched at 2026-10-09T03:05:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (8)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (6)
Date (UTC)Associated PRProw job
PR #74412108512930990395392
PR #71622108511709156085760
PR #74372108374328641851392
PR #73052108214686754279424
PR #71432107875990943305728
PR #74002107875420446658560
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert61.93%6 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 6: 5 · Oct 7: 2 · Oct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 0none
Full failure examples (3)
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:37:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0004864-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-vk9xs", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-vk9xs has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:42:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0004864-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-64b5859f7f-5glfr", region="westus3", severity="warning"
  Description: Pod sessiongate/sessiongate-64b5859f7f-5glfr has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:42:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0004864-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-76c4b4c8bc-8gvkr", region="westus3", severity="warning"
  Description: Pod fleet/fleet-controller-76c4b4c8bc-8gvkr has been in a non-ready state for longer than 5 minutes.
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1804928-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14292482-vmss000002-dskt8", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-14292482-vmss000002-dskt8 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1804928-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14292482-vmss000000-472fv", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-14292482-vmss000000-472fv has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:31:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1804928-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dmqmp", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-dmqmp has been in a non-ready state for longer than 5 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-23980774-vmss000000-slpqz", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-23980774-vmss000000-slpqz has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-lzv6r", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-lzv6r has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-qbqjf", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-qbqjf has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:12:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-6f6gx", region="westus3", severity="warning"
  Description: Pod swift-recorder/swift-recorder-6f6gx has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #74392109581128020004864
PR #71622109578665191804928
PR #73972108558761344897024
PR #74332108258997998981120
PR #73052107961335642853376
PR #73972107647281405628416
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e61.93%6 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 1 · Oct 12: 2 · Oct 13: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:412]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x8ceb7456d30>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:412]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x8ceb7456d30>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23913a80e730>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23913a80e730>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.006s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23a94d2a72e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.006s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23a94d2a72e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.201
Affected runs (6)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
PR #74672109660658009116672
PR #74572109370651696435200
PR #72922108438042090934272
PR #74142108377968924430336
PR #70972107680144293367808
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e61.93%6 of 311 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 6: 1 · Oct 7: 0 · Oct 8: 2 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 0INT
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x2f223f7aa0d8>: 
    resource group "hcp-metrics-lh6cn9z9mpvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-lh6cn9z9mpvb" to finish deleting: GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "name": "666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "status": "Failed",
      "startTime": "2026-10-12T13:00:52.055384192Z",
      "endTime": "2026-10-12T13:24:57.118445449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td8hkede6pik7afuli4rqulo2mdm0ib still exists (deletion dispatched at 2026-10-12T13:00:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
            correlation ID: 32f74568-8662-4b4e-8805-1aefdc453b8a
            client request ID: 470d44cf-4554-4545-aed1-856e45d54cdb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x2f223f7aa0d8>: 
    resource group "hcp-metrics-lh6cn9z9mpvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-lh6cn9z9mpvb" to finish deleting: GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "name": "666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "status": "Failed",
      "startTime": "2026-10-12T13:00:52.055384192Z",
      "endTime": "2026-10-12T13:24:57.118445449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td8hkede6pik7afuli4rqulo2mdm0ib still exists (deletion dispatched at 2026-10-12T13:00:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
            correlation ID: 32f74568-8662-4b4e-8805-1aefdc453b8a
            client request ID: 470d44cf-4554-4545-aed1-856e45d54cdb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xde424e4090>: 
    resource group "admin-credential-lifecycle-test-7p8rt54bzxxl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-cm6t76" in resourcegroup="admin-credential-lifecycle-test-7p8rt54bzxxl" to finish deleting: GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad",
      "name": "d217aaa4-6911-41fe-a395-709b350055ad",
      "status": "Failed",
      "startTime": "2026-10-12T12:20:06.890936008Z",
      "endTime": "2026-10-12T12:44:10.847761888Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td7rmojjop6e5ni0mk3h0r2c95v0aqq still exists (deletion dispatched at 2026-10-12T12:20:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
            correlation ID: 5a65ad43-439a-49ac-a5b3-3ea36ef8c6fd
            client request ID: 3b679075-36d2-4af2-ad13-7fdd7af0d4bb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xde424e4090>: 
    resource group "admin-credential-lifecycle-test-7p8rt54bzxxl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-cm6t76" in resourcegroup="admin-credential-lifecycle-test-7p8rt54bzxxl" to finish deleting: GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad",
      "name": "d217aaa4-6911-41fe-a395-709b350055ad",
      "status": "Failed",
      "startTime": "2026-10-12T12:20:06.890936008Z",
      "endTime": "2026-10-12T12:44:10.847761888Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td7rmojjop6e5ni0mk3h0r2c95v0aqq still exists (deletion dispatched at 2026-10-12T12:20:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
            correlation ID: 5a65ad43-439a-49ac-a5b3-3ea36ef8c6fd
            client request ID: 3b679075-36d2-4af2-ad13-7fdd7af0d4bb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126a68f88048>: 
    resource group "nodepool-update-nodes-ppzw4fscb65q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ppzw4fscb65q" to finish deleting: GET https://rp.j7543168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d",
      "name": "5402a6f6-777b-405f-bf42-17f6202e880d",
      "status": "Failed",
      "startTime": "2026-10-09T09:00:29.162024664Z",
      "endTime": "2026-10-09T09:24:36.108080611Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb5gs6ackm9riurrcuq3gadk7un22q7 still exists (deletion dispatched at 2026-10-09T09:00:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126a68f88048>: 
    resource group "nodepool-update-nodes-ppzw4fscb65q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ppzw4fscb65q" to finish deleting: GET https://rp.j7543168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d",
      "name": "5402a6f6-777b-405f-bf42-17f6202e880d",
      "status": "Failed",
      "startTime": "2026-10-09T09:00:29.162024664Z",
      "endTime": "2026-10-09T09:24:36.108080611Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb5gs6ackm9riurrcuq3gadk7un22q7 still exists (deletion dispatched at 2026-10-09T09:00:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
Affected runs (6)
Date (UTC)Associated PRProw job
PR #74722109611255084879872
PR #74682109600459105767424
PR #74392108458935697543168
PR #71052108409414854643712
PR #71802108317092523544576
PR #73252108228889548427264
alert [svc] MaestroAgentMetricsProxyDown fired
full failure pattern:
alert [svc] MaestroAgentMetricsProxyDown fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
alert61.93%6 of 311 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 6: 2 · Oct 7: 1 · Oct 8: 2 · Oct 9: 3 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:02:04Z
  Ended:       2026-10-09T15:12:09Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4897024-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T14:26:58Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5137536-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T03:29:55Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j9299456-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #73972108558761344897024
PR #67462108552522435137536
PR #73942108385613169299456
PR #74142108199393621446656
PR #74272108191395574452224
PR #73972107647281405628416
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
full failure pattern:
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
alert51.61%5 of 311 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 4 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 4 · Oct 13: 0none
Full failure examples (3)
alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9114496/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j9114496' could not be found."
  }
}
--------------------------------------------------------------------------------
alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9114496/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j9114496' could not be found."
  }
}
--------------------------------------------------------------------------------
alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2837760' could not be found."
  }
}
--------------------------------------------------------------------------------
Contributing tests (2)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert collection is complete5
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert collection is complete5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #74392109792838379114496
PR #69202109727639982837760
PR #72262108491565306482688
PR #74222108246398544646144
PR #72262108169795277950976
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e51.61%5 of 311 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 6: 3 · Oct 7: 0 · Oct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 3 · Oct 13: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xe01e7de22c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2x2v84z9v765" to finish creating: GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565",
      "name": "3d382676-ff60-4e1f-babd-b92045c21565",
      "status": "Failed",
      "startTime": "2026-10-12T19:22:34.907448575Z",
      "endTime": "2026-10-12T19:51:42.771361832Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
            correlation ID: 7441ce1a-f7e5-42c1-bf5b-99ce537aeb48
            client request ID: fed3af2b-11b1-43b5-8577-26857bd2a8c1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xe01e7de22c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2x2v84z9v765" to finish creating: GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565",
      "name": "3d382676-ff60-4e1f-babd-b92045c21565",
      "status": "Failed",
      "startTime": "2026-10-12T19:22:34.907448575Z",
      "endTime": "2026-10-12T19:51:42.771361832Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
            correlation ID: 7441ce1a-f7e5-42c1-bf5b-99ce537aeb48
            client request ID: fed3af2b-11b1-43b5-8577-26857bd2a8c1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x329d818e85e0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-7hcwvl89hlds" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa",
      "name": "af485567-16d3-466b-8cef-6a0e9bce52fa",
      "status": "Failed",
      "startTime": "2026-10-12T12:42:57.954228784Z",
      "endTime": "2026-10-12T13:11:58.34288805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
            correlation ID: 70b5741c-ea80-4b9a-9e33-ac7a70af1ce7
            client request ID: f176988a-7ecb-4ecf-8063-4ee5d27bb712
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x329d818e85e0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-7hcwvl89hlds" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa",
      "name": "af485567-16d3-466b-8cef-6a0e9bce52fa",
      "status": "Failed",
      "startTime": "2026-10-12T12:42:57.954228784Z",
      "endTime": "2026-10-12T13:11:58.34288805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
            correlation ID: 70b5741c-ea80-4b9a-9e33-ac7a70af1ce7
            client request ID: f176988a-7ecb-4ecf-8063-4ee5d27bb712
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x3d2ba8a26f40>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-tmxkg7m275vm" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301",
      "name": "06ab8b07-36db-4738-9b5d-4d16092f8301",
      "status": "Failed",
      "startTime": "2026-10-12T09:26:40.450447729Z",
      "endTime": "2026-10-12T09:55:40.068785838Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
            correlation ID: c5b7b9a4-edc0-484e-9068-a103f50c2ab8
            client request ID: d0e69db4-dd9a-407e-93a4-3206c81ac5a6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x3d2ba8a26f40>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-tmxkg7m275vm" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301",
      "name": "06ab8b07-36db-4738-9b5d-4d16092f8301",
      "status": "Failed",
      "startTime": "2026-10-12T09:26:40.450447729Z",
      "endTime": "2026-10-12T09:55:40.068785838Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
            correlation ID: c5b7b9a4-edc0-484e-9068-a103f50c2ab8
            client request ID: d0e69db4-dd9a-407e-93a4-3206c81ac5a6
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (5)
Date (UTC)Associated PRProw job
PR #74792109715328622661632
PR #74702109611047785598976
PR #74632109563550279143424
PR #72272108589017825021952
PR #74142108055963964542976
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e51.61%5 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-mpwdzx"
Unexpected error:
    <*fmt.wrapError | 0x149c10e309a0>: 
    DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x149c10e30940>{
            msg: "DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-mpwdzx"
Unexpected error:
    <*fmt.wrapError | 0x149c10e309a0>: 
    DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x149c10e30940>{
            msg: "DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-s9x9xg"
Unexpected error:
    <*fmt.wrapError | 0x1d3dff18b020>: 
    DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1d3dff18afe0>{
            msg: "DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-s9x9xg"
Unexpected error:
    <*fmt.wrapError | 0x1d3dff18b020>: 
    DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1d3dff18afe0>{
            msg: "DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-j7bk7p"
Unexpected error:
    <*fmt.wrapError | 0x76fa41aa1e0>: 
    DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x76fa41aa1a0>{
            msg: "DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-j7bk7p"
Unexpected error:
    <*fmt.wrapError | 0x76fa41aa1e0>: 
    DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x76fa41aa1a0>{
            msg: "DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (5)
Date (UTC)Associated PRProw job
PR #72372109549126109630464
PR #74512109538389626720256
PR #72272108648932484583424
PR #74372108452755680006144
PR #73972107751278686769152
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
full failure pattern:
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
alert51.61%5 of 311 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 2 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 0none
Full failure examples (3)
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9114496: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9114496/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j9114496' could not be found."
  }
}
--------------------------------------------------------------------------------
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2837760' could not be found."
  }
}
--------------------------------------------------------------------------------
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert collection is complete5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #74392109792838379114496
PR #69202109727639982837760
PR #72262108491565306482688
PR #74222108246398544646144
PR #72262108169795277950976
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
alert41.29%4 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsOct 6: 2 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 3 · Oct 13: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:53:49Z
  Ended:       2026-10-12T20:09:48Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cdb52", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T19:53:49Z
  Ended:       2026-10-12T20:13:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cdb52", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:06:14Z
  Ended:       2026-10-12T13:34:15Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5f8cm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:06:14Z
  Ended:       2026-10-12T13:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5f8cm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T09:49:03Z
  Ended:       2026-10-12T10:13:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4hsb2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T09:49:03Z
  Ended:       2026-10-12T10:17:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4hsb2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74792109715328622661632
PR #74702109611047785598976
PR #74632109563550279143424
PR #74142108055963964542976
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e30.96%3 of 311 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsOct 6: 1 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1 · Oct 12: 1 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x23adf9021e40>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-92gd76g6d97g" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
            correlation ID: 349f9734-994e-46db-888e-e1caf77e8fb4
            client request ID: 46549edd-5c88-4d3d-b17b-df9c20ad793d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x23adf9021e40>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-92gd76g6d97g" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
            correlation ID: 349f9734-994e-46db-888e-e1caf77e8fb4
            client request ID: 46549edd-5c88-4d3d-b17b-df9c20ad793d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zhszcgnzg7zb/cluster-candidate-4-22-gwkf7b should provision
Unexpected error:
    <*fmt.wrapError | 0x17457dc5e0e0>: 
    failed to create HCP cluster cluster-candidate-4-22-gwkf7b: failed waiting for cluster="cluster-candidate-4-22-gwkf7b" in resourcegroup="rg-candidate-4-22-zhszcgnzg7zb" to finish creating: GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
            correlation ID: e870b595-733c-4e50-b159-c06646eb16ba
            client request ID: 7c347d72-8099-4796-893d-59762507c230
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zhszcgnzg7zb/cluster-candidate-4-22-gwkf7b should provision
Unexpected error:
    <*fmt.wrapError | 0x17457dc5e0e0>: 
    failed to create HCP cluster cluster-candidate-4-22-gwkf7b: failed waiting for cluster="cluster-candidate-4-22-gwkf7b" in resourcegroup="rg-candidate-4-22-zhszcgnzg7zb" to finish creating: GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
            correlation ID: e870b595-733c-4e50-b159-c06646eb16ba
            client request ID: 7c347d72-8099-4796-893d-59762507c230
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng"
Unexpected error:
    <*fmt.wrapError | 0x1dd12c4c3a80>: 
    failed to create NodePool np-ft6nng: failed waiting for nodepool="np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng" in resourcegroup="rg-candidate-4-22-j25t2jm5j66l" to finish creating: GET https://rp.j9390848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng"
Unexpected error:
    <*fmt.wrapError | 0x1dd12c4c3a80>: 
    failed to create NodePool np-ft6nng: failed waiting for nodepool="np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng" in resourcegroup="rg-candidate-4-22-j25t2jm5j66l" to finish creating: GET https://rp.j9390848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #74402109536551896616960
PR #74242109420238498959360
PR #73832108217820029390848
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error
e2e10.32%1 of 311 job runs affectedNoiseSignal: Noise — source deserialization/no-output errorShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (1)
77f90 sp=0x3ae32ce77ed0 pc=0x586c752
net/http.(*Transport).startDialConnForLocked.func1()
	/usr/lib/golang/src/net/http/transport.go:1629 +0x35 fp=0x3ae32ce77fe0 sp=0x3ae32ce77f90 pc=0x586c595
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32ce77fe8 sp=0x3ae32ce77fe0 pc=0x5540381
created by net/http.(*Transport).startDialConnForLocked in goroutine 119
	/usr/lib/golang/src/net/http/transport.go:1628 +0x112

goroutine 167 gp=0x3ae32ce014a0 m=nil [semacquire]:
runtime.gopark(0x3f00400063cb02?, 0x3ae32da24402?, 0xe0?, 0xc0?, 0x3ae32d696bb8?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x3ae32d6969c0 sp=0x3ae32d6969a0 pc=0x553818e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x16f798a4, 0x0, 0x0, 0x0, 0xd)
	/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x3ae32d696a28 sp=0x3ae32d6969c0 pc=0x5513872
runtime.semacquire(...)
	/usr/lib/golang/src/runtime/sema.go:143
runtime.gcMarkDone()
	/usr/lib/golang/src/runtime/mgc.go:1018 +0x32 fp=0x3ae32d696ac8 sp=0x3ae32d696a28 pc=0x54d6232
runtime.gcAssistAlloc(0x3ae32ce014a0)
	/usr/lib/golang/src/runtime/mgcmark.go:647 +0x2f6 fp=0x3ae32d696b58 sp=0x3ae32d696ac8 pc=0x54dadd6
runtime.deductAssistCredit(0x1?)
	/usr/lib/golang/src/runtime/malloc.go:1861 +0x46 fp=0x3ae32d696b78 sp=0x3ae32d696b58 pc=0x54cb6a6
runtime.mallocgc(0x590, 0xf965760, 0x1)
	/usr/lib/golang/src/runtime/malloc.go:1113 +0x67 fp=0x3ae32d696ba8 sp=0x3ae32d696b78 pc=0x5534e67
runtime.newobject(0x3ae32da2445d?)
	/usr/lib/golang/src/runtime/malloc.go:2157 +0x25 fp=0x3ae32d696bd0 sp=0x3ae32d696ba8 pc=0x54cb7a5
crypto/x509.parseCertificate({0x3ae32da4c000, 0x56d, 0x582})
	/usr/lib/golang/src/crypto/x509/parser.go:887 +0x45 fp=0x3ae32d697068 sp=0x3ae32d696bd0 pc=0x5729285
crypto/x509.ParseCertificate({0x3ae32da4c000?, 0x56d, 0x2ea83?})
	/usr/lib/golang/src/crypto/x509/parser.go:1077 +0x1d fp=0x3ae32d697090 sp=0x3ae32d697068 pc=0x572a69d
crypto/x509.(*CertPool).AppendCertsFromPEM(0x3ae32d4196e0, {0x3ae32da80000?, 0x100a51a0?, 0x10059470?})
	/usr/lib/golang/src/crypto/x509/cert_pool.go:231 +0x125 fp=0x3ae32d6971b0 sp=0x3ae32d697090 pc=0x571d2c5
crypto/x509.loadSystemRoots()
	/usr/lib/golang/src/crypto/x509/root_unix.go:72 +0x391 fp=0x3ae32d697298 sp=0x3ae32d6971b0 pc=0x572ccf1
crypto/x509.initSystemRoots()
	/usr/lib/golang/src/crypto/x509/root.go:43 +0x65 fp=0x3ae32d6972d8 sp=0x3ae32d697298 pc=0x572c785
sync.(*Once).doSlow(0x54ca9ea?, 0x1f1a89e3ef160b9?)
	/usr/lib/golang/src/sync/once.go:78 +0xac fp=0x3ae32d697330 sp=0x3ae32d6972d8 pc=0x555198c
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:69
crypto/x509.systemRootsPool()
	/usr/lib/golang/src/crypto/x509/root.go:32 +0x45 fp=0x3ae32d697380 sp=0x3ae32d697330 pc=0x572c645
crypto/x509.(*Certificate).Verify(0x3ae32d4ae008, {{0x3ae32d1d9be0, 0x19}, 0x3ae32d419620, 0x0, {0xc2aa7a68520d364e, 0xb6db3cc, 0x16f76d00}, {0x0, 0x0, ...}, ...})
	/usr/lib/golang/src/crypto/x509/verify.go:578 +0x18b fp=0x3ae32d697530 sp=0x3ae32d697380 pc=0x572f5cb
crypto/tls.(*Conn).verifyServerCertificate(0x3ae32d432008, {0x3ae32d49a5a0, 0x3, 0x4})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:1168 +0xb0e fp=0x3ae32d6978b8 sp=0x3ae32d697530 pc=0x5781dee
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:640 +0x29a fp=0x3ae32d697ab0 sp=0x3ae32d6978b8 pc=0x578629a
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:141 +0x7cc fp=0x3ae32d697b70 sp=0x3ae32d697ab0 pc=0x578344c
crypto/tls.(*Conn).clientHandshake(0x3ae32d432008, {0x1018a380, 0x3ae32ca24690})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x3ae32d697d88 sp=0x3ae32d697b70 pc=0x577cb68
crypto/tls.(*Conn).clientHandshake-fm({0x1018a380?, 0x3ae32ca24690?})
	<autogenerated>:1 +0x2a fp=0x3ae32d697db0 sp=0x3ae32d697d88 pc=0x57bcaca
crypto/tls.(*Conn).handshakeContext(0x3ae32d432008, {0x1018a380, 0x3ae32c9f4460})
	/usr/lib/golang/src/crypto/tls/conn.go:1562 +0x329 fp=0x3ae32d697f70 sp=0x3ae32d697db0 pc=0x5773829
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/golang/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
	/usr/lib/golang/src/net/http/transport.go:1736 +0x6e fp=0x3ae32d697fe0 sp=0x3ae32d697f70 pc=0x586d3ce
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32d697fe8 sp=0x3ae32d697fe0 pc=0x5540381
created by net/http.(*persistConn).addTLS in goroutine 120
	/usr/lib/golang/src/net/http/transport.go:1732 +0x309

rax    0x0
rbx    0x7fe37a988640
rcx    0x7fe3c265d94c
rdx    0x6
rdi    0xa6f0
rsi    0xa6f4
rbp    0xa6f4
rsp    0x7fe37a987790
r8     0x7fe37a987860
r9     0x7fe3c27834e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7fe37a987b60
r14    0x3ae32bf97860
r15    0xffffffffffffffff
rip    0x7fe3c265d94c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
77f90 sp=0x3ae32ce77ed0 pc=0x586c752
net/http.(*Transport).startDialConnForLocked.func1()
	/usr/lib/golang/src/net/http/transport.go:1629 +0x35 fp=0x3ae32ce77fe0 sp=0x3ae32ce77f90 pc=0x586c595
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32ce77fe8 sp=0x3ae32ce77fe0 pc=0x5540381
created by net/http.(*Transport).startDialConnForLocked in goroutine 119
	/usr/lib/golang/src/net/http/transport.go:1628 +0x112

goroutine 167 gp=0x3ae32ce014a0 m=nil [semacquire]:
runtime.gopark(0x3f00400063cb02?, 0x3ae32da24402?, 0xe0?, 0xc0?, 0x3ae32d696bb8?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x3ae32d6969c0 sp=0x3ae32d6969a0 pc=0x553818e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x16f798a4, 0x0, 0x0, 0x0, 0xd)
	/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x3ae32d696a28 sp=0x3ae32d6969c0 pc=0x5513872
runtime.semacquire(...)
	/usr/lib/golang/src/runtime/sema.go:143
runtime.gcMarkDone()
	/usr/lib/golang/src/runtime/mgc.go:1018 +0x32 fp=0x3ae32d696ac8 sp=0x3ae32d696a28 pc=0x54d6232
runtime.gcAssistAlloc(0x3ae32ce014a0)
	/usr/lib/golang/src/runtime/mgcmark.go:647 +0x2f6 fp=0x3ae32d696b58 sp=0x3ae32d696ac8 pc=0x54dadd6
runtime.deductAssistCredit(0x1?)
	/usr/lib/golang/src/runtime/malloc.go:1861 +0x46 fp=0x3ae32d696b78 sp=0x3ae32d696b58 pc=0x54cb6a6
runtime.mallocgc(0x590, 0xf965760, 0x1)
	/usr/lib/golang/src/runtime/malloc.go:1113 +0x67 fp=0x3ae32d696ba8 sp=0x3ae32d696b78 pc=0x5534e67
runtime.newobject(0x3ae32da2445d?)
	/usr/lib/golang/src/runtime/malloc.go:2157 +0x25 fp=0x3ae32d696bd0 sp=0x3ae32d696ba8 pc=0x54cb7a5
crypto/x509.parseCertificate({0x3ae32da4c000, 0x56d, 0x582})
	/usr/lib/golang/src/crypto/x509/parser.go:887 +0x45 fp=0x3ae32d697068 sp=0x3ae32d696bd0 pc=0x5729285
crypto/x509.ParseCertificate({0x3ae32da4c000?, 0x56d, 0x2ea83?})
	/usr/lib/golang/src/crypto/x509/parser.go:1077 +0x1d fp=0x3ae32d697090 sp=0x3ae32d697068 pc=0x572a69d
crypto/x509.(*CertPool).AppendCertsFromPEM(0x3ae32d4196e0, {0x3ae32da80000?, 0x100a51a0?, 0x10059470?})
	/usr/lib/golang/src/crypto/x509/cert_pool.go:231 +0x125 fp=0x3ae32d6971b0 sp=0x3ae32d697090 pc=0x571d2c5
crypto/x509.loadSystemRoots()
	/usr/lib/golang/src/crypto/x509/root_unix.go:72 +0x391 fp=0x3ae32d697298 sp=0x3ae32d6971b0 pc=0x572ccf1
crypto/x509.initSystemRoots()
	/usr/lib/golang/src/crypto/x509/root.go:43 +0x65 fp=0x3ae32d6972d8 sp=0x3ae32d697298 pc=0x572c785
sync.(*Once).doSlow(0x54ca9ea?, 0x1f1a89e3ef160b9?)
	/usr/lib/golang/src/sync/once.go:78 +0xac fp=0x3ae32d697330 sp=0x3ae32d6972d8 pc=0x555198c
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:69
crypto/x509.systemRootsPool()
	/usr/lib/golang/src/crypto/x509/root.go:32 +0x45 fp=0x3ae32d697380 sp=0x3ae32d697330 pc=0x572c645
crypto/x509.(*Certificate).Verify(0x3ae32d4ae008, {{0x3ae32d1d9be0, 0x19}, 0x3ae32d419620, 0x0, {0xc2aa7a68520d364e, 0xb6db3cc, 0x16f76d00}, {0x0, 0x0, ...}, ...})
	/usr/lib/golang/src/crypto/x509/verify.go:578 +0x18b fp=0x3ae32d697530 sp=0x3ae32d697380 pc=0x572f5cb
crypto/tls.(*Conn).verifyServerCertificate(0x3ae32d432008, {0x3ae32d49a5a0, 0x3, 0x4})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:1168 +0xb0e fp=0x3ae32d6978b8 sp=0x3ae32d697530 pc=0x5781dee
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:640 +0x29a fp=0x3ae32d697ab0 sp=0x3ae32d6978b8 pc=0x578629a
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:141 +0x7cc fp=0x3ae32d697b70 sp=0x3ae32d697ab0 pc=0x578344c
crypto/tls.(*Conn).clientHandshake(0x3ae32d432008, {0x1018a380, 0x3ae32ca24690})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x3ae32d697d88 sp=0x3ae32d697b70 pc=0x577cb68
crypto/tls.(*Conn).clientHandshake-fm({0x1018a380?, 0x3ae32ca24690?})
	<autogenerated>:1 +0x2a fp=0x3ae32d697db0 sp=0x3ae32d697d88 pc=0x57bcaca
crypto/tls.(*Conn).handshakeContext(0x3ae32d432008, {0x1018a380, 0x3ae32c9f4460})
	/usr/lib/golang/src/crypto/tls/conn.go:1562 +0x329 fp=0x3ae32d697f70 sp=0x3ae32d697db0 pc=0x5773829
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/golang/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
	/usr/lib/golang/src/net/http/transport.go:1736 +0x6e fp=0x3ae32d697fe0 sp=0x3ae32d697f70 pc=0x586d3ce
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32d697fe8 sp=0x3ae32d697fe0 pc=0x5540381
created by net/http.(*persistConn).addTLS in goroutine 120
	/usr/lib/golang/src/net/http/transport.go:1732 +0x309

rax    0x0
rbx    0x7fe37a988640
rcx    0x7fe3c265d94c
rdx    0x6
rdi    0xa6f0
rsi    0xa6f4
rbp    0xa6f4
rsp    0x7fe37a987790
r8     0x7fe37a987860
r9     0x7fe3c27834e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7fe37a987b60
r14    0x3ae32bf97860
r15    0xffffffffffffffff
rip    0x7fe3c265d94c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74462108673315018117120
ERROR CODE: DeadlineExceeded; provider Microsoft.Network
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Network
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e103.22%10 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 6: 57 · Oct 7: 23 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3c574774040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-m842v74xwlhh" to finish creating: GET https://rp.j3880064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "name": "310e7732-14f1-4d4c-820c-fb521ce69b9b",
      "status": "Failed",
      "startTime": "2026-10-07T18:10:29.82492819Z",
      "endTime": "2026-10-07T18:39:31.852478192Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-2tvj4-977r6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-977r6 to exist\nMachine v20261001-np-1-2tvj4-xn4zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-2tvj4-xn4zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-2tvj4-977r6: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-m842v74xwlhh--managed/v20261001-np-1-2tvj4-977r6-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-m842v74xwlhh--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-2tvj4-977r6-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb"
Unexpected error:
    <*fmt.wrapError | 0x2d8b7da68c20>: 
    failed to create NodePool np-74nbzb: failed waiting for nodepool="np-74nbzb" for cluster "cluster-candidate-4-23-74nbzb" in resourcegroup="rg-candidate-4-23-z987ndt4v67c" to finish creating: GET https://rp.j6440704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "name": "ba9ec6af-5a25-490d-a829-97ac1ecf6a0e",
      "status": "Failed",
      "startTime": "2026-10-07T18:12:58.862762914Z",
      "endTime": "2026-10-07T18:41:59.529725798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97 to exist\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Compute/virtualMachines/y2i2u2o9d4w2c0t-np-74nbzb-r688d-k8q7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z987ndt4v67c-candidate-74nbzb--managed/providers/Microsoft.Network/networkInterfaces/y2i2u2o9d4w2c0t-np-74nbzb-r688d-22d97-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '2' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2af3a314160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-gvbd6v968qsn" to finish creating: GET https://rp.j8893056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "name": "0f0eaa3c-f753-4064-9736-bc1f28a4ed88",
      "status": "Failed",
      "startTime": "2026-10-07T17:57:49.24099524Z",
      "endTime": "2026-10-07T18:26:50.588266562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-vh9lt-cmt4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-cmt4b to exist\nMachine v20261001-np-1-vh9lt-j5d2x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-vh9lt-j5d2x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-vh9lt-cmt4b: Failed: interfaces failed to create or update. err: failed to get existing resource v20261001preview-gvbd6v968qsn--managed/v20261001-np-1-vh9lt-cmt4b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-gvbd6v968qsn--managed/providers/Microsoft.Network/networkInterfaces/v20261001-np-1-vh9lt-cmt4b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (11)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.235
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.12
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (10)
Date (UTC)Associated PRProw job
PR #72482107883389313880064
PR #68072107880778506440704
PR #74142107879778638893056
PR #74002107875420446658560
PR #74122107873993380859904
PR #74042107834229550223360
PR #63642107828301232869376
PR #73972107751278686769152
PR #73992107750792218808320
PR #73712107750686056779776
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e72.25%7 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 6: 2 · Oct 7: 1 · Oct 8: 2 · Oct 9: 1 · Oct 10: 1 · Oct 11: 1 · Oct 12: 1 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-kxb95g" for cluster "cluster-hs-pre-kxb95g"
Unexpected error:
    <*fmt.wrapError | 0xb6d00db1440>: 
    failed to create NodePool np-kxb95g: failed waiting for nodepool="np-kxb95g" for cluster "cluster-hs-pre-kxb95g" in resourcegroup="rg-hs-pre--x6fpqw6hqtjb" to finish creating: GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "name": "dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "status": "Failed",
      "startTime": "2026-10-12T19:32:07.510834189Z",
      "endTime": "2026-10-12T20:01:15.571433072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
            correlation ID: bfbdef2c-b02a-48b8-a68a-a2439acfc443
            client request ID: 7e134c20-557c-48a5-9100-cbcf934467d5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-kxb95g" for cluster "cluster-hs-pre-kxb95g"
Unexpected error:
    <*fmt.wrapError | 0xb6d00db1440>: 
    failed to create NodePool np-kxb95g: failed waiting for nodepool="np-kxb95g" for cluster "cluster-hs-pre-kxb95g" in resourcegroup="rg-hs-pre--x6fpqw6hqtjb" to finish creating: GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "name": "dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "status": "Failed",
      "startTime": "2026-10-12T19:32:07.510834189Z",
      "endTime": "2026-10-12T20:01:15.571433072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
            correlation ID: bfbdef2c-b02a-48b8-a68a-a2439acfc443
            client request ID: 7e134c20-557c-48a5-9100-cbcf934467d5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1b569c9d54e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlcxj9z6srvl" to finish creating: GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04",
      "name": "5a974196-cc3e-4c19-83bb-20557d537e04",
      "status": "Failed",
      "startTime": "2026-10-12T00:41:24.479383887Z",
      "endTime": "2026-10-12T01:10:29.269134441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
            correlation ID: 2bffc39b-cd83-4954-a0a9-c7daa45e4979
            client request ID: 8e77b332-cff0-4849-90bf-1e7ad363de44
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1b569c9d54e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlcxj9z6srvl" to finish creating: GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04",
      "name": "5a974196-cc3e-4c19-83bb-20557d537e04",
      "status": "Failed",
      "startTime": "2026-10-12T00:41:24.479383887Z",
      "endTime": "2026-10-12T01:10:29.269134441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
            correlation ID: 2bffc39b-cd83-4954-a0a9-c7daa45e4979
            client request ID: 8e77b332-cff0-4849-90bf-1e7ad363de44
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x5b5e959eb80>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hxz5lhgx6pgc" to finish creating: GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "name": "bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "status": "Failed",
      "startTime": "2026-10-10T09:08:14.887674935Z",
      "endTime": "2026-10-10T09:37:18.731959559Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
            correlation ID: 85344fe9-db3b-408b-ba74-a3aed4f6938b
            client request ID: 900c9a16-ab81-46bf-aba5-3f67f267761a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x5b5e959eb80>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hxz5lhgx6pgc" to finish creating: GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "name": "bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "status": "Failed",
      "startTime": "2026-10-10T09:08:14.887674935Z",
      "endTime": "2026-10-10T09:37:18.731959559Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
            correlation ID: 85344fe9-db3b-408b-ba74-a3aed4f6938b
            client request ID: 900c9a16-ab81-46bf-aba5-3f67f267761a
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (7)
Date (UTC)Associated PRProw job
PR #73732109708565542342656
PR #74392109426221556174848
PR #72272108830741738557440
PR #74372108452755680006144
PR #74032108182743610298368
PR #73612108025089978011648
PR #73102107872947002675200
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e72.25%7 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsOct 6: 0 · Oct 7: 2 · Oct 8: 2 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x72d88398300>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2m7pgwml79qx": PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
            correlation ID: 0548989b-5739-4134-ae92-44e0b94372c4
            client request ID: eff3cdde-c6fa-4be6-bf0f-91fa006cb430
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x72d88398300>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2m7pgwml79qx": PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
            correlation ID: 0548989b-5739-4134-ae92-44e0b94372c4
            client request ID: eff3cdde-c6fa-4be6-bf0f-91fa006cb430
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #73972109643872534532096
PR #73972108513285430054912
PR #73972108452968792592384
PR #73972108316538086887424
PR #73972108178486299463680
PR #73972107751278686769152
PR #73972107681070840287232
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
full failure pattern:
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e72.25%7 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 4 · Oct 8: 3 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xed922aab58>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0xed9195f1e0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0xed91fd22c0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xed922aab58>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0xed9195f1e0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0xed91fd22c0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x7b70a422fc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x7b70a4e87a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x7b70a6fbd20>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x7b70a422fc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x7b70a4e87a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x7b70a6fbd20>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e29e0f51bc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e29e2b846a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e29e176e4f0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e29e0f51bc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e29e2b846a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e29e176e4f0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #73252108228889548427264
PR #73252108090608831696896
PR #73252108018145892503552
PR #73252107970662109810688
PR #73252107916955439599616
PR #74072107818144989974528
PR #74002107676117312737280
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e51.61%5 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 3 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3475813fe160>: 
    DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3475813fe0e0>{
            msg: "DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3475813fe160>: 
    DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3475813fe0e0>{
            msg: "DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x247e648e4440>: 
    DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x247e648e4420>{
            msg: "DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x247e648e4440>: 
    DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x247e648e4420>{
            msg: "DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-ff6jdg"
Unexpected error:
    <*fmt.wrapError | 0x17d44c09e520>: 
    DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x17d44c09e500>{
            msg: "DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-ff6jdg"
Unexpected error:
    <*fmt.wrapError | 0x17d44c09e520>: 
    DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x17d44c09e500>{
            msg: "DNS for agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7dms2.apps.aro.i7t6k9j8s5g2b2z.byrq.j6670976.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (5)
Date (UTC)Associated PRProw job
PR #71652108676067857272832
PR #74392108648873588166656
PR #74172107967978476670976
PR #73102107739528360890368
PR #74022107717187903950848
alert [hcp] HCPClusterOperatorUnavailable fired
full failure pattern:
alert [hcp] HCPClusterOperatorUnavailable fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
alert51.61%5 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 2 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 3 · Oct 13: 0none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T21:30:33Z
  Ended:       2026-10-12T22:07:32Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j6043392-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdf8lncsu04rjm64fbreumbi4l47bg3-l5g9b5r3v2p1b0s", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tdf8lncsu04rjm64fbreumbi4l47bg3-l5g9b5r3v2p1b0s (management cluster ci01-j6043392-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:50:18Z
  Ended:       2026-10-12T20:25:22Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2503936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tddoajh9tanmcldflt6gegfe8au9a4r-a8g2t7k0z6k5s9l", region="centralus", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tddoajh9tanmcldflt6gegfe8au9a4r-a8g2t7k0z6k5s9l (management cluster ci01-j2503936-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:45:35Z
  Ended:       2026-10-12T20:14:37Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j5649024-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tddn1vgtlc9nve2f6icfn7fqenmjbmm-x2t6l2r1k1q6d2z", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tddn1vgtlc9nve2f6icfn7fqenmjbmm-x2t6l2r1k1q6d2z (management cluster ci01-j5649024-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #74722109729919306043392
PR #74702109706138382503936
PR #74712109705158475649024
PR #74072107818144989974528
PR #74002107676117312737280
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36
full failure pattern:
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e51.61%5 of 311 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 5 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-k9v4gs" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-k9v4gs" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-zx5s4v" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-zx5s4v" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.009s.
cluster "sre-version-pin-5gfztc" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.009s.
cluster "sre-version-pin-5gfztc" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE advances a control plane to an SRE-pinned version5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #71622109754183480512512
PR #71622109675831700754432
PR #73832109662114334380032
PR #71622109629712874606592
PR #73832109600098458537984
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderCluster...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e41.29%4 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 4 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1d94d9c80a8>: 
    resource group "autoscaling-cluster-tj5rdtq5zbg6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tj5rdtq5zbg6" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373",
      "name": "962a68bc-b321-487b-b276-789b2307a373",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:48.91579633Z",
      "endTime": "2026-10-09T12:22:56.421671364Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1d94d9c80a8>: 
    resource group "autoscaling-cluster-tj5rdtq5zbg6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tj5rdtq5zbg6" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373",
      "name": "962a68bc-b321-487b-b276-789b2307a373",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:48.91579633Z",
      "endTime": "2026-10-09T12:22:56.421671364Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3346ee20e4b0>: 
    resource group "feature-aggregation-vk8bg8lddcq2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-vk8bg8lddcq2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964",
      "name": "4724c843-31f5-4e70-a53a-a9dc39567964",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:11.356025969Z",
      "endTime": "2026-10-09T12:29:15.890678807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3346ee20e4b0>: 
    resource group "feature-aggregation-vk8bg8lddcq2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-vk8bg8lddcq2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964",
      "name": "4724c843-31f5-4e70-a53a-a9dc39567964",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:11.356025969Z",
      "endTime": "2026-10-09T12:29:15.890678807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x10e954cac0c0>: 
    resource group "fips-enabled-g648xvh6nzj9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g648xvh6nzj9" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "name": "6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:00.540461838Z",
      "endTime": "2026-10-09T12:16:05.840029374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x10e954cac0c0>: 
    resource group "fips-enabled-g648xvh6nzj9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g648xvh6nzj9" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "name": "6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:00.540461838Z",
      "endTime": "2026-10-09T12:16:05.840029374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74412108512930990395392
PR #71622108511709156085760
PR #74372108374328641851392
PR #71432107875990943305728
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...
full failure pattern:
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e41.29%4 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 1 · Oct 11: 0 · Oct 12: 3 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x6f1edda03a0>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-jdq4zztv647q" to finish creating: GET https://rp.j2613760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/97c24699-e347-4966-b180-bf6ae5ff9b03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/97c24699-e347-4966-b180-bf6ae5ff9b03",
      "name": "97c24699-e347-4966-b180-bf6ae5ff9b03",
      "status": "Failed",
      "startTime": "2026-10-12T21:54:28.61846625Z",
      "endTime": "2026-10-12T21:54:42.231408957Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jdq4zztv647q/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2613760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/97c24699-e347-4966-b180-bf6ae5ff9b03
            correlation ID: 8f0a4fda-224a-436d-84e3-d641535231f1
            client request ID: d44e41d0-633f-4992-a25b-ce6473619f8a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x6f1edda03a0>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-jdq4zztv647q" to finish creating: GET https://rp.j2613760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/97c24699-e347-4966-b180-bf6ae5ff9b03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/97c24699-e347-4966-b180-bf6ae5ff9b03",
      "name": "97c24699-e347-4966-b180-bf6ae5ff9b03",
      "status": "Failed",
      "startTime": "2026-10-12T21:54:28.61846625Z",
      "endTime": "2026-10-12T21:54:42.231408957Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jdq4zztv647q/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2613760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/97c24699-e347-4966-b180-bf6ae5ff9b03
            correlation ID: 8f0a4fda-224a-436d-84e3-d641535231f1
            client request ID: d44e41d0-633f-4992-a25b-ce6473619f8a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x10139db40a0>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-v52tck7hh95p" to finish creating: GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "name": "567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "status": "Failed",
      "startTime": "2026-10-12T21:36:04.986032041Z",
      "endTime": "2026-10-12T21:36:44.722884546Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-v52tck7hh95p/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'ccbf438f-08ec-4e2b-ae4e-219ce7dc3762' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
            correlation ID: 7bddac41-f6a3-4695-b95b-13972ea0840e
            client request ID: ab9e631f-f159-4b76-a0df-2b299f43d7d8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x10139db40a0>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-v52tck7hh95p" to finish creating: GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "name": "567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "status": "Failed",
      "startTime": "2026-10-12T21:36:04.986032041Z",
      "endTime": "2026-10-12T21:36:44.722884546Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-v52tck7hh95p/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'ccbf438f-08ec-4e2b-ae4e-219ce7dc3762' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
            correlation ID: 7bddac41-f6a3-4695-b95b-13972ea0840e
            client request ID: ab9e631f-f159-4b76-a0df-2b299f43d7d8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0x32753cabb000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lcmmtbwtg46t" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "name": "b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "status": "Failed",
      "startTime": "2026-10-12T09:39:05.150325854Z",
      "endTime": "2026-10-12T09:39:46.941270418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
            correlation ID: 84029b8a-bf49-4092-9e8c-07ae2562e0b2
            client request ID: 168cc3ec-3b47-4824-95c6-847f6de09edb
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0x32753cabb000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lcmmtbwtg46t" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "name": "b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "status": "Failed",
      "startTime": "2026-10-12T09:39:05.150325854Z",
      "endTime": "2026-10-12T09:39:46.941270418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
            correlation ID: 84029b8a-bf49-4092-9e8c-07ae2562e0b2
            client request ID: 168cc3ec-3b47-4824-95c6-847f6de09edb
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74282109749870842613760
PR #73972109742847518314496
PR #74632109563550279143424
PR #73742108886756110635008
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.R...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e30.96%3 of 311 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 12 · Oct 13: 0none
Full failure examples (3)
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should get an existing HCP cluster and report a missing cluster through the RP3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelImmutable cluster configuration rejects an immutable cluster DNS prefix PATCH without changing the shared cluster3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelImmutable node pool configuration rejects an immutable node pool OS disk size PATCH without changing the shared node pool3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelExternal auth issuer validation rejects an invalid external auth issuer URL without leaving a child or changing the shared cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNode pool create validation rejects a negative replica count without creating a node pool or changing the shared cluster1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #74712109616618123300864
PR #74722109611255084879872
PR #74702109611047785598976
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e30.96%3 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 2 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x151218b082b8>: 
    resource group "rg-cluster-back-version-9554sthg8hnz": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-9554sthg8hnz", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x151216d07800>{
                msg: "resource group \"rg-cluster-back-version-9554sthg8hnz\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x151216d077a0>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x1512170c3950>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x151218b082b8>: 
    resource group "rg-cluster-back-version-9554sthg8hnz": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-9554sthg8hnz", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x151216d07800>{
                msg: "resource group \"rg-cluster-back-version-9554sthg8hnz\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x151216d077a0>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x1512170c3950>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x213e82b661e0>: 
    resource group "rg-cluster-back-version-s9llrx7qtffw": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-s9llrx7qtffw", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x213e826ac820>{
                msg: "resource group \"rg-cluster-back-version-s9llrx7qtffw\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x213e826ac800>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x213e83e111d0>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x213e82b661e0>: 
    resource group "rg-cluster-back-version-s9llrx7qtffw": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-s9llrx7qtffw", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-s9llrx7qtffw" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x213e826ac820>{
                msg: "resource group \"rg-cluster-back-version-s9llrx7qtffw\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x213e826ac800>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x213e83e111d0>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x213e83e11170>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-s9llrx7qtffw\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x213e831e7270>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-s9llrx7qtffw",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c28962ca078>: 
    resource group "rg-cluster-back-version-kcpxjfv4fpw8": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c289468e340>{
                msg: "resource group \"rg-cluster-back-version-kcpxjfv4fpw8\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x3c289468e320>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x3c2894a68660>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c28962ca078>: 
    resource group "rg-cluster-back-version-kcpxjfv4fpw8": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kcpxjfv4fpw8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c289468e340>{
                msg: "resource group \"rg-cluster-back-version-kcpxjfv4fpw8\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x3c289468e320>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x3c2894a68660>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x3c2894a68630>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kcpxjfv4fpw8\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x3c2894fa1480>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kcpxjfv4fpw8",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.203
Affected runs (3)
Date (UTC)Associated PRProw job
PR #69202109658742915076096
PR #74072107818144989974528
PR #74002107676117312737280
VerifyHyperShiftIngressCertificate timed out after 10m0
full failure pattern:
VerifyHyperShiftIngressCertificate timed out after 10m0
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e30.96%3 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 2 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x5336d4f4280>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.008s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x5336d4f4220>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x5336d4f4180>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should serve the default ingress certificate and allow a customer certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for 5.13
Affected runs (3)
Date (UTC)Associated PRProw job
PR #73252108090608831696896
PR #73252108018145892503552
PR #73252107970662109810688
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e20.64%2 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 135 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f8378088558>: 
    resource group "kusto-logs-zdcz6ngd7l8s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zdcz6ngd7l8s" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "name": "ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "status": "Failed",
      "startTime": "2026-10-07T13:55:49.637158017Z",
      "endTime": "2026-10-07T14:19:49.807578144Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vu3tto21eb5vs2oeuv4d035r4kh9q still exists (deletion dispatched at 2026-10-07T13:56:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f8378088558>: 
    resource group "kusto-logs-zdcz6ngd7l8s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zdcz6ngd7l8s" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "name": "ca8af611-54d1-469b-80b1-afacd1e9f0c2",
      "status": "Failed",
      "startTime": "2026-10-07T13:55:49.637158017Z",
      "endTime": "2026-10-07T14:19:49.807578144Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vu3tto21eb5vs2oeuv4d035r4kh9q still exists (deletion dispatched at 2026-10-07T13:56:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xf860122e060>: 
    resource group "rg-zstream-upgrade-4-21-d7rnngth6rmt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-s5wrnm" in resourcegroup="rg-zstream-upgrade-4-21-d7rnngth6rmt" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "name": "de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "status": "Failed",
      "startTime": "2026-10-07T14:14:15.375309734Z",
      "endTime": "2026-10-07T14:38:20.14126941Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtq0dj9uuqrc354smigdjo5gckvsa still exists (deletion dispatched at 2026-10-07T14:14:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xf860122e060>: 
    resource group "rg-zstream-upgrade-4-21-d7rnngth6rmt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-s5wrnm" in resourcegroup="rg-zstream-upgrade-4-21-d7rnngth6rmt" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "name": "de2aa0e7-6bfa-4ba6-91bf-20123b9953f5",
      "status": "Failed",
      "startTime": "2026-10-07T14:14:15.375309734Z",
      "endTime": "2026-10-07T14:38:20.14126941Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtq0dj9uuqrc354smigdjo5gckvsa still exists (deletion dispatched at 2026-10-07T14:14:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3d715b6dc0c0>: 
    resource group "admin-api-serialconsole-kgtb4656t7wx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-kgtb4656t7wx" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "name": "63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "status": "Failed",
      "startTime": "2026-10-07T14:05:26.627856798Z",
      "endTime": "2026-10-07T14:29:30.238184303Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtu5p75nk86f54m2uds6vreosd2p2 still exists (deletion dispatched at 2026-10-07T14:05:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3d715b6dc0c0>: 
    resource group "admin-api-serialconsole-kgtb4656t7wx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-kgtb4656t7wx" to finish deleting: GET https://rp.j9974528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "name": "63b430ac-4aff-4ba1-9a60-4787a227b94b",
      "status": "Failed",
      "startTime": "2026-10-07T14:05:26.627856798Z",
      "endTime": "2026-10-07T14:29:30.238184303Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9vtu5p75nk86f54m2uds6vreosd2p2 still exists (deletion dispatched at 2026-10-07T14:05:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (68)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.232
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.12
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.21 minor to 4.22 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.232
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74072107818144989974528
PR #74002107676117312737280
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e20.64%2 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x14de9654a318>: 
    resource group "arm64-vm-cluster-bbmplmj7xnjh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bbmplmj7xnjh" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "name": "8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "status": "Failed",
      "startTime": "2026-10-09T12:10:18.071494213Z",
      "endTime": "2026-10-09T12:34:22.504221789Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x14de9654a318>: 
    resource group "arm64-vm-cluster-bbmplmj7xnjh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bbmplmj7xnjh" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "name": "8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "status": "Failed",
      "startTime": "2026-10-09T12:10:18.071494213Z",
      "endTime": "2026-10-09T12:34:22.504221789Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xdad5eeb8138>: 
    resource group "external-auth-cluster-cmj7tlkjf2f4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-cmj7tlkjf2f4" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e",
      "name": "a7900d0f-9832-43df-824c-157a3931804e",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:18.997805773Z",
      "endTime": "2026-10-09T12:35:21.856574986Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xdad5eeb8138>: 
    resource group "external-auth-cluster-cmj7tlkjf2f4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-cmj7tlkjf2f4" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e",
      "name": "a7900d0f-9832-43df-824c-157a3931804e",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:18.997805773Z",
      "endTime": "2026-10-09T12:35:21.856574986Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3ed0c46420d8>: 
    resource group "rg-negative-tests-z8rjxzl5wzq5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8rjxzl5wzq5" to finish deleting: GET https://rp.j8408192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "name": "92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "status": "Failed",
      "startTime": "2026-10-07T20:31:10.664805555Z",
      "endTime": "2026-10-07T20:55:12.515323239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3ed0c46420d8>: 
    resource group "rg-negative-tests-z8rjxzl5wzq5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8rjxzl5wzq5" to finish deleting: GET https://rp.j8408192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "name": "92268962-7674-4c53-a6ec-a9b7d3b7b795",
      "status": "Failed",
      "startTime": "2026-10-07T20:31:10.664805555Z",
      "endTime": "2026-10-07T20:55:12.515323239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73972108513285430054912
PR #72882107904235558408192
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e20.64%2 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x31481464c780>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vxxh5xhm4hm8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vxxh5xhm4hm8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x314814c9ae20>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x31481464c780>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vxxh5xhm4hm8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vxxh5xhm4hm8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x314814c9ae20>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x13eb0e2191a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bkn7fgqj44mp" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-bkn7fgqj44mp\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x13eb0e00ddf0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x13eb0e2191a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bkn7fgqj44mp" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-bkn7fgqj44mp\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x13eb0e00ddf0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapErrors | 0x7f2f57b95f0>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-hwcbq22lzw9l" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x7f2f606c4c0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
            },
            <*fmt.wrapErrors | 0x7f2f57b95c0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x7f2f606c4c0>{
                        s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapErrors | 0x7f2f57b95f0>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-hwcbq22lzw9l" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x7f2f606c4c0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
            },
            <*fmt.wrapErrors | 0x7f2f57b95c0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x7f2f606c4c0>{
                        s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
PR #74402108607415241936896
alert [svc] FrontendPathLatency fired
full failure pattern:
alert [svc] FrontendPathLatency fired
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
alert20.64%2 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T10:08:14Z
  Ended:       2026-10-09T10:15:16Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4483328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4483328-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-09T10:31:13Z
  Ended:       2026-10-09T10:40:15Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4483328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4483328-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:31:34Z
  Ended:       2026-10-09T00:04:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4990592-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j4990592-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69442108480044224483328
PR #73642108317114874990592
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e20.64%2 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x2ec259084840>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-tv5qjjvw6mt5" to finish creating: GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16",
      "name": "639bbef1-ebbd-4367-896d-cec3c0263d16",
      "status": "Failed",
      "startTime": "2026-10-12T15:42:56.655438798Z",
      "endTime": "2026-10-12T15:43:02.11596193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
            correlation ID: 1ceff82d-31f8-4b03-ac69-9b0cd1c3b903
            client request ID: 24f2f909-e5ab-41c2-8601-8e5d6aea83f8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x2ec259084840>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-tv5qjjvw6mt5" to finish creating: GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16",
      "name": "639bbef1-ebbd-4367-896d-cec3c0263d16",
      "status": "Failed",
      "startTime": "2026-10-12T15:42:56.655438798Z",
      "endTime": "2026-10-12T15:43:02.11596193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
            correlation ID: 1ceff82d-31f8-4b03-ac69-9b0cd1c3b903
            client request ID: 24f2f909-e5ab-41c2-8601-8e5d6aea83f8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x18aecfdf0aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dq2w5s6mnvnb" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "name": "5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "status": "Failed",
      "startTime": "2026-10-08T08:17:53.86568203Z",
      "endTime": "2026-10-08T08:18:02.050800903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x18aecfdf0aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dq2w5s6mnvnb" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "name": "5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "status": "Failed",
      "startTime": "2026-10-08T08:17:53.86568203Z",
      "endTime": "2026-10-08T08:18:02.050800903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74402109655959428141056
PR #74122108092712069632000
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e20.64%2 of 311 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
PR #74402108607415241936896

Environment: INT

Failure patterns: 2 · Failures matched: 7 · Total runs: 10 · Runs affected: 3

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e330.00%3 of 10 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 1 · Oct 7: 0 · Oct 8: 6 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25f97d5e2360>: 
    resource group "tls-endpoint-cluster-z5gqw56sgjhd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-z5gqw56sgjhd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "name": "cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "status": "Failed",
      "startTime": "2026-10-08T21:40:36.777928134Z",
      "endTime": "2026-10-08T22:04:36.806096142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tarq2rh1nt9l8n2qiphckhb46c0kvk6 still exists (deletion dispatched at 2026-10-08T21:40:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25f97d5e2360>: 
    resource group "tls-endpoint-cluster-z5gqw56sgjhd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-z5gqw56sgjhd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "name": "cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "status": "Failed",
      "startTime": "2026-10-08T21:40:36.777928134Z",
      "endTime": "2026-10-08T22:04:36.806096142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tarq2rh1nt9l8n2qiphckhb46c0kvk6 still exists (deletion dispatched at 2026-10-08T21:40:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x63eda6241c8>: 
    resource group "rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "name": "c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:08.268369953Z",
      "endTime": "2026-10-08T17:27:11.496076999Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmuj8loas31r4p690qgf675h7c53q still exists (deletion dispatched at 2026-10-08T17:03:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x63eda6241c8>: 
    resource group "rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "name": "c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:08.268369953Z",
      "endTime": "2026-10-08T17:27:11.496076999Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmuj8loas31r4p690qgf675h7c53q still exists (deletion dispatched at 2026-10-08T17:03:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x317d01c40258>: 
    resource group "v20260901preview-9cxtwb94v2tg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-9cxtwb94v2tg" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "name": "4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:49.176345278Z",
      "endTime": "2026-10-08T17:27:51.502199148Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmtqk63qrfodamveh24r6lp7p43kt still exists (deletion dispatched at 2026-10-08T17:03:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x317d01c40258>: 
    resource group "v20260901preview-9cxtwb94v2tg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-9cxtwb94v2tg" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "name": "4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:49.176345278Z",
      "endTime": "2026-10-08T17:27:51.502199148Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmtqk63qrfodamveh24r6lp7p43kt still exists (deletion dispatched at 2026-10-08T17:03:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2108289457667969024
n/a2108221811563958272
n/a2107988105985265664
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e110.00%1 of 10 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:148]: failed to delete customer resource group "cx-rg-cluster-9528856dtms2"
Unexpected error:
    <*fmt.wrapErrors | 0x2df35190540>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded
    {
        msg: "failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2df35ff14c0>{
                s: "timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:148]: failed to delete customer resource group "cx-rg-cluster-9528856dtms2"
Unexpected error:
    <*fmt.wrapErrors | 0x2df35190540>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded
    {
        msg: "failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2df35ff14c0>{
                s: "timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108221811563958272

Environment: PROD

Failure patterns: 22 · Failures matched: 274 · Total runs: 45 · Runs affected: 21

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e920.00%9 of 45 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 6: 6 · Oct 7: 43 · Oct 8: 36 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c"
Unexpected error:
    <*fmt.wrapError | 0x553d3fef80>: 
    failed to create NodePool np-kn7h4c: failed waiting for nodepool="np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c" in resourcegroup="rg-candidate-5-0-5tp4pjp8hk9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "name": "b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "status": "Failed",
      "startTime": "2026-10-08T18:41:12.213982763Z",
      "endTime": "2026-10-08T19:10:15.712133624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c"
Unexpected error:
    <*fmt.wrapError | 0x553d3fef80>: 
    failed to create NodePool np-kn7h4c: failed waiting for nodepool="np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c" in resourcegroup="rg-candidate-5-0-5tp4pjp8hk9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "name": "b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "status": "Failed",
      "startTime": "2026-10-08T18:41:12.213982763Z",
      "endTime": "2026-10-08T19:10:15.712133624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n"
Unexpected error:
    <*fmt.wrapError | 0x35ae7eca0680>: 
    failed to create NodePool np-ntb88n: failed waiting for nodepool="np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n" in resourcegroup="rg-candidate-5-1-tvlhq8pw8r2j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122",
      "name": "00bc71ab-da38-456c-8cc5-ad615c84b122",
      "status": "Failed",
      "startTime": "2026-10-08T18:45:34.339197477Z",
      "endTime": "2026-10-08T19:14:37.105782951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n"
Unexpected error:
    <*fmt.wrapError | 0x35ae7eca0680>: 
    failed to create NodePool np-ntb88n: failed waiting for nodepool="np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n" in resourcegroup="rg-candidate-5-1-tvlhq8pw8r2j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122",
      "name": "00bc71ab-da38-456c-8cc5-ad615c84b122",
      "status": "Failed",
      "startTime": "2026-10-08T18:45:34.339197477Z",
      "endTime": "2026-10-08T19:14:37.105782951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm"
Unexpected error:
    <*fmt.wrapError | 0x3108b67364e0>: 
    failed to create NodePool np-8mqkrm: failed waiting for nodepool="np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm" in resourcegroup="rg-candidate-4-20-xmx872mb68nv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "name": "827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "status": "Failed",
      "startTime": "2026-10-08T18:32:58.020047255Z",
      "endTime": "2026-10-08T19:02:05.605513431Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm"
Unexpected error:
    <*fmt.wrapError | 0x3108b67364e0>: 
    failed to create NodePool np-8mqkrm: failed waiting for nodepool="np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm" in resourcegroup="rg-candidate-4-20-xmx872mb68nv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "name": "827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "status": "Failed",
      "startTime": "2026-10-08T18:32:58.020047255Z",
      "endTime": "2026-10-08T19:02:05.605513431Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (27)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.206
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.216
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.226
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.236
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.06
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.16
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API6
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public6
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.232
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.12
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (9)
Date (UTC)Associated PRProw job
n/a2108248148466995200
n/a2108193764467019776
n/a2108041306176491520
n/a2108014673302392832
n/a2107977652647235584
n/a2107969374202630144
n/a2107917271417491456
n/a2107872150416265216
n/a2107652362557460480
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e715.56%7 of 45 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 6: 1 · Oct 7: 5 · Oct 8: 4 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2f509ecfe3c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bw4mbhgcm827" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "name": "d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "status": "Failed",
      "startTime": "2026-10-08T18:07:59.549473703Z",
      "endTime": "2026-10-08T18:37:05.482678031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2f509ecfe3c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bw4mbhgcm827" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "name": "d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "status": "Failed",
      "startTime": "2026-10-08T18:07:59.549473703Z",
      "endTime": "2026-10-08T18:37:05.482678031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3c50440e0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nkr4c2vp869x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e",
      "name": "5de3f7d0-158f-4a7e-a885-89292635394e",
      "status": "Failed",
      "startTime": "2026-10-08T18:08:57.323157264Z",
      "endTime": "2026-10-08T18:38:05.479695963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3c50440e0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nkr4c2vp869x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e",
      "name": "5de3f7d0-158f-4a7e-a885-89292635394e",
      "status": "Failed",
      "startTime": "2026-10-08T18:08:57.323157264Z",
      "endTime": "2026-10-08T18:38:05.479695963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x6335433a090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-tpwrtzjcnmjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "name": "3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:33.571990767Z",
      "endTime": "2026-10-08T14:56:35.04864797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-8swjn-292k4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-8swjn-292k4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x6335433a090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-tpwrtzjcnmjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "name": "3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:33.571990767Z",
      "endTime": "2026-10-08T14:56:35.04864797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-8swjn-292k4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-8swjn-292k4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2108248148466995200
n/a2108193764467019776
n/a2108014673302392832
n/a2107977652647235584
n/a2107917271417491456
n/a2107872150416265216
n/a2107652362557460480
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
e2e613.33%6 of 45 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsOct 6: 2 · Oct 7: 2 · Oct 8: 5 · Oct 9: 5 · Oct 10: 5 · Oct 11: 4 · Oct 12: 4 · Oct 13: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2a4f44e2d020>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gwxj7rx975nt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gwxj7rx975nt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 28bc07db-2472-48a0-b8ec-f57404d49246
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2a4f44e2d020>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gwxj7rx975nt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gwxj7rx975nt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 28bc07db-2472-48a0-b8ec-f57404d49246
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xb63c5f25ec0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ffrc4jvplr5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ffrc4jvplr5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: 85fbf18c-d76e-44a1-a17c-93587336d09b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xb63c5f25ec0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ffrc4jvplr5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ffrc4jvplr5w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: 85fbf18c-d76e-44a1-a17c-93587336d09b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3b32f7ffab10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-n86b7vqhp6m8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-n86b7vqhp6m8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: 04e10847-e793-4a67-b207-a326ae8ddcfc
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3b32f7ffab10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-n86b7vqhp6m8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-n86b7vqhp6m8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: 04e10847-e793-4a67-b207-a326ae8ddcfc
            client request ID: unavailable
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request4
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2109781178750865408
n/a2109418786846150656
n/a2109056606359523328
n/a2108694214656135168
n/a2108331766317060096
n/a2107969374202630144
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenSh...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e48.89%4 of 45 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 6: 18 · Oct 7: 17 · Oct 8: 10 · Oct 9: 10 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-s865cq" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x29e047bb9ce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-84fl6x5gkgjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-s865cq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-s865cq" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x29e047bb9ce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-84fl6x5gkgjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-s865cq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-pwmrlt" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x364694577140>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bcxf7z6jvkdm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-pwmrlt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-pwmrlt" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x364694577140>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bcxf7z6jvkdm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-pwmrlt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qldg5l" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x200557060510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-hldpsj9c8292/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qldg5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qldg5l" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x200557060510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-hldpsj9c8292/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qldg5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (19)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2108377065311965184
n/a2108014673302392832
n/a2107969374202630144
n/a2107652362557460480
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenS...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e48.89%4 of 45 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 6: 12 · Oct 7: 12 · Oct 8: 9 · Oct 9: 9 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6f0cf085080>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6zf46c" resourcegroup="rg-np-dg-minor-6zf46c-hll6hfr44s9d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6zf46c-hll6hfr44s9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6zf46c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6f0cf085080>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6zf46c" resourcegroup="rg-np-dg-minor-6zf46c-hll6hfr44s9d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6zf46c-hll6hfr44s9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6zf46c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1508ab4d7940>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lc4jfj" resourcegroup="rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lc4jfj/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1508ab4d7940>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lc4jfj" resourcegroup="rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lc4jfj/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x22aa54096dc0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-qs6slk" resourcegroup="rg-np-skip-minor-qs6slk-svmgzjwq29hn": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qs6slk-svmgzjwq29hn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qs6slk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x22aa54096dc0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-qs6slk" resourcegroup="rg-np-skip-minor-qs6slk-svmgzjwq29hn": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qs6slk-svmgzjwq29hn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qs6slk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (12)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to update node pool labels and taints3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.203
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2108377065311965184
n/a2108014673302392832
n/a2107969374202630144
n/a2107652362557460480
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpe...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e36.67%3 of 45 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 6: 2 · Oct 7: 4 · Oct 8: 3 · Oct 9: 3 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-g5xfmb" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x2d844c21d830>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5xdvnklmwfq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-g5xfmb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-g5xfmb" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x2d844c21d830>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5xdvnklmwfq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-g5xfmb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-t4k2nv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x9a9211d2ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wdg9f8phhv6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-t4k2nv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-t4k2nv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x9a9211d2ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wdg9f8phhv6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-t4k2nv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-kqzff9" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xe01aaa2bbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7pmn5ng5wrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kqzff9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-kqzff9" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xe01aaa2bbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7pmn5ng5wrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kqzff9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.233
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.03
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2108377065311965184
n/a2108014673302392832
n/a2107652362557460480
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
full failure pattern:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e36.67%3 of 45 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 0 · Oct 10: 1 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:175]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
    <*errors.joinError | 0x2d289f100030>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2d289e7b3500>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2d289e7b34e0>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2d289f0dae60>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2d289e6a6040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2d289e6a6020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2d289e168dc0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:175]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
    <*errors.joinError | 0x2d289f100030>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2d289e7b3500>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2d289e7b34e0>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2d289f0dae60>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2d289e6a6040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2d289e6a6020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2d289e168dc0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x3ac58974db78>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3ac589d3d2a0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589d3d280>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3ac589190be0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3ac589e5f3e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589e5f3c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3ac58a59a640>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x3ac58974db78>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3ac589d3d2a0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589d3d280>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3ac589190be0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3ac589e5f3e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589e5f3c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3ac58a59a640>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x2e5cf51ae018>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2e5cf5120ca0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf5120c80>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4986960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2e5cf52a2040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf52a2020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4b086e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x2e5cf51ae018>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2e5cf5120ca0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf5120c80>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4986960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2e5cf52a2040>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2e5cf52a2020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2e5cf4b086e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2109781178750865408
n/a2109056606359523328
n/a2107969374202630144
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e817.78%8 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 5 · Oct 8: 3 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a6cffe3f060>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jct5pzjh8mxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "name": "b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "status": "Failed",
      "startTime": "2026-10-08T15:48:06.632742492Z",
      "endTime": "2026-10-08T16:17:11.778582607Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a6cffe3f060>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jct5pzjh8mxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "name": "b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "status": "Failed",
      "startTime": "2026-10-08T15:48:06.632742492Z",
      "endTime": "2026-10-08T16:17:11.778582607Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x12d67242a00>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdb7hgvdsm8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "name": "1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "status": "Failed",
      "startTime": "2026-10-08T05:42:55.903350034Z",
      "endTime": "2026-10-08T06:12:00.742498982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x12d67242a00>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdb7hgvdsm8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "name": "1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "status": "Failed",
      "startTime": "2026-10-08T05:42:55.903350034Z",
      "endTime": "2026-10-08T06:12:00.742498982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2523ecbc0ea0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zgfh95c7gz2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "name": "cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "status": "Failed",
      "startTime": "2026-10-08T03:28:30.373260167Z",
      "endTime": "2026-10-08T03:57:35.369426461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2523ecbc0ea0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zgfh95c7gz2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "name": "cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "status": "Failed",
      "startTime": "2026-10-08T03:28:30.373260167Z",
      "endTime": "2026-10-08T03:57:35.369426461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to delete a nodepool2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool1
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2108193764467019776
n/a2108041306176491520
n/a2108014673302392832
n/a2107977652647235584
n/a2107969374202630144
n/a2107917271417491456
n/a2107872150416265216
n/a2107652362557460480
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube con...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 51 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z9xhpt9dxztr/cluster-candidate-4-22-8frl8d should provision
Unexpected error:
    <*fmt.wrapError | 0xc3c4040eec0>: 
    failed to create HCP cluster cluster-candidate-4-22-8frl8d: failed waiting for cluster="cluster-candidate-4-22-8frl8d" in resourcegroup="rg-candidate-4-22-z9xhpt9dxztr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "name": "6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.12350471Z",
      "endTime": "2026-10-07T11:13:04.942313802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-z9xhpt9dxztr/cluster-candidate-4-22-8frl8d should provision
Unexpected error:
    <*fmt.wrapError | 0xc3c4040eec0>: 
    failed to create HCP cluster cluster-candidate-4-22-8frl8d: failed waiting for cluster="cluster-candidate-4-22-8frl8d" in resourcegroup="rg-candidate-4-22-z9xhpt9dxztr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "name": "6731dbc8-8cd4-41ae-bf5b-f0e9ab07cfbf",
      "status": "Failed",
      "startTime": "2026-10-07T10:44:02.12350471Z",
      "endTime": "2026-10-07T11:13:04.942313802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2851e862a620>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ws854cfjbfc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "name": "5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:27.70826866Z",
      "endTime": "2026-10-07T12:10:34.84968826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2851e862a620>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ws854cfjbfc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "name": "5d35187f-9481-4680-bc88-1fa2a5bd4bb4",
      "status": "Failed",
      "startTime": "2026-10-07T11:41:27.70826866Z",
      "endTime": "2026-10-07T12:10:34.84968826Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x231265410c40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-m6pl2gptcdfp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "name": "3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:27.398564047Z",
      "endTime": "2026-10-07T12:08:34.791319585Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x231265410c40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-m6pl2gptcdfp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "name": "3147c124-12ef-4d26-acf9-96afbf8f7fdc",
      "status": "Failed",
      "startTime": "2026-10-07T11:39:27.398564047Z",
      "endTime": "2026-10-07T12:08:34.791319585Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (51)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 7 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x186c1f59fe00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-xkrgtz49hrh4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xkrgtz49hrh4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x186c1f59fe00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-xkrgtz49hrh4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xkrgtz49hrh4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x5e92b4ffb60>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-jsrm64gc6pjs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-jsrm64gc6pjs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x5e92b4ffb60>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-jsrm64gc6pjs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-jsrm64gc6pjs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0x1d4613307440>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-xccktj7qbpzs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0x1d4613307440>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-xccktj7qbpzs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 6: 3 · Oct 7: 3 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf"
Unexpected error:
    <*fmt.wrapError | 0x312a48ea2980>: 
    failed to create NodePool np-zgtwtf: failed starting nodepool creation "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf" in resourcegroup="rg-nightly-4-20-j42q4dvn7t74": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-j42q4dvn7t74/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zgtwtf/nodePools/np-zgtwtf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf"
Unexpected error:
    <*fmt.wrapError | 0x312a48ea2980>: 
    failed to create NodePool np-zgtwtf: failed starting nodepool creation "np-zgtwtf" for cluster "cluster-nightly-4-20-zgtwtf" in resourcegroup="rg-nightly-4-20-j42q4dvn7t74": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-j42q4dvn7t74/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zgtwtf/nodePools/np-zgtwtf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x32c6fec798e0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-vbjww8" in resourcegroup="rg-np-skip-minor-vbjww8-9dlhkbhpd9hb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vbjww8-9dlhkbhpd9hb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vbjww8/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x32c6fec798e0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-vbjww8" in resourcegroup="rg-np-skip-minor-vbjww8-9dlhkbhpd9hb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vbjww8-9dlhkbhpd9hb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vbjww8/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1829881ca3e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7pmxvz" in resourcegroup="rg-np-version-upgrade-7pmxvz-jjjscx2plvvj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pmxvz-jjjscx2plvvj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pmxvz/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-07-010648
Unexpected error:
    <*fmt.wrapError | 0x1829881ca3e0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7pmxvz" in resourcegroup="rg-np-version-upgrade-7pmxvz-jjjscx2plvvj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pmxvz-jjjscx2plvvj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pmxvz/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-07-010648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107969374202630144
VerifyBasicAccess failed: failed to list services: Unauthorized
full failure pattern:
VerifyBasicAccess failed: failed to list services: Unauthorized
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 6: 1 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 2 · Oct 12: 0 · Oct 13: 0STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x11944cc8930>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x11944de5c00>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x11944de5be0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x11945431e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x11944978520>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x11944978500>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x11944c97ae0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x11944cc8930>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x11944de5c00>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x11944de5be0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x11945431e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x11944978520>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x11944978500>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x11944c97ae0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
    <*errors.joinError | 0x19c6c82ad0e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x19c6c78ffea0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c78ffe80>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7ea8960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x19c6c8695740>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c8695720>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7b65c20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:137]: failed to verify basic cluster health for idms-e2e-hcp-cluster
Unexpected error:
    <*errors.joinError | 0x19c6c82ad0e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x19c6c78ffea0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c78ffe80>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7ea8960>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x19c6c8695740>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x19c6c8695720>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x19c6c7b65c20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109418786846150656
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ptqdgd" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x21d67b55ea20>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-ptqdgd: failed waiting for cluster="cp-ystream-upgrade-5-0-ptqdgd" in resourcegroup="rg-cp-ystream-upgrade-5-0-dznstqd64m54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044",
      "name": "9d14066e-1629-48f8-aa68-f556c4fd7044",
      "status": "Failed",
      "startTime": "2026-10-12T07:51:24.545231929Z",
      "endTime": "2026-10-12T08:20:29.935309867Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
            correlation ID: 7a60ad0c-37f6-45ef-98dd-efdfad687d6c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ptqdgd" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x21d67b55ea20>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-ptqdgd: failed waiting for cluster="cp-ystream-upgrade-5-0-ptqdgd" in resourcegroup="rg-cp-ystream-upgrade-5-0-dznstqd64m54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044",
      "name": "9d14066e-1629-48f8-aa68-f556c4fd7044",
      "status": "Failed",
      "startTime": "2026-10-12T07:51:24.545231929Z",
      "endTime": "2026-10-12T08:20:29.935309867Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
            correlation ID: 7a60ad0c-37f6-45ef-98dd-efdfad687d6c
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109535177171537920
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-q7jjp7" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x298b49702ce0>: 
    failed to create HCP cluster cluster-zstream-4-21-q7jjp7: failed waiting for cluster="cluster-zstream-4-21-q7jjp7" in resourcegroup="rg-zstream-upgrade-4-21-jd6h7xk8btv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "name": "c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "status": "Failed",
      "startTime": "2026-10-12T14:02:09.836072198Z",
      "endTime": "2026-10-12T14:31:10.061357256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, packageserver, cluster-network-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
            correlation ID: 414509d7-91c9-4911-b025-fd41c0e067b1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-q7jjp7" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x298b49702ce0>: 
    failed to create HCP cluster cluster-zstream-4-21-q7jjp7: failed waiting for cluster="cluster-zstream-4-21-q7jjp7" in resourcegroup="rg-zstream-upgrade-4-21-jd6h7xk8btv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "name": "c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "status": "Failed",
      "startTime": "2026-10-12T14:02:09.836072198Z",
      "endTime": "2026-10-12T14:31:10.061357256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, packageserver, cluster-network-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
            correlation ID: 414509d7-91c9-4911-b025-fd41c0e067b1
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109630653069791232
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x151c97ead140>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-p86tvztrm7hd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "name": "01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "status": "Failed",
      "startTime": "2026-10-12T01:36:23.272806003Z",
      "endTime": "2026-10-12T02:05:32.513113167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
            correlation ID: e3c8027d-936a-4cfa-802c-eac5497d3369
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x151c97ead140>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-p86tvztrm7hd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "name": "01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "status": "Failed",
      "startTime": "2026-10-12T01:36:23.272806003Z",
      "endTime": "2026-10-12T02:05:32.513113167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
            correlation ID: e3c8027d-936a-4cfa-802c-eac5497d3369
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109435185735929856
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x38da2adfeb00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-kbw6v2d2bhz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095",
      "name": "4899c274-4471-4ab9-871b-22368aa84095",
      "status": "Failed",
      "startTime": "2026-10-12T00:35:15.167750325Z",
      "endTime": "2026-10-12T01:04:20.403308122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-fk5wl-zdgdp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-fk5wl-zdgdp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
            correlation ID: 79b16636-2655-4af8-9ac0-f8e4eda0ad35
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x38da2adfeb00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-kbw6v2d2bhz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095",
      "name": "4899c274-4471-4ab9-871b-22368aa84095",
      "status": "Failed",
      "startTime": "2026-10-12T00:35:15.167750325Z",
      "endTime": "2026-10-12T01:04:20.403308122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-fk5wl-zdgdp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-fk5wl-zdgdp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
            correlation ID: 79b16636-2655-4af8-9ac0-f8e4eda0ad35
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109435185735929856
ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/user...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.; provider Microsoft.ManagedIdentity
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:101]: failed to deploy dedicated managed identities
Unexpected error:
    <*fmt.wrapError | 0x2a6c3876d160>: 
    failed waiting for deployment "mi-missing-mi-hcp-cluster-8qb82d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-missing-mi-hcp-cluster-8qb82d/operationStatuses/08584101764787793835
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterNonMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:101]: failed to deploy dedicated managed identities
Unexpected error:
    <*fmt.wrapError | 0x2a6c3876d160>: 
    failed waiting for deployment "mi-missing-mi-hcp-cluster-8qb82d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-missing-mi-hcp-cluster-8qb82d/operationStatuses/08584101764787793835
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterNonMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108014673302392832
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1 · Oct 12: 0 · Oct 13: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x5325624e738>: 
    resource group "cni-cilium-g76pf6t9h842": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-g76pf6t9h842" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84",
      "name": "179cc551-21d3-4583-b554-f672e37b3a84",
      "status": "Failed",
      "startTime": "2026-10-11T22:50:53.504973281Z",
      "endTime": "2026-10-11T23:14:57.954117538Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tcruuiers9mtsh6kof4evdrtdk9p3e4 still exists (deletion dispatched at 2026-10-11T22:50:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x5325624e738>: 
    resource group "cni-cilium-g76pf6t9h842": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-g76pf6t9h842" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/179cc551-21d3-4583-b554-f672e37b3a84",
      "name": "179cc551-21d3-4583-b554-f672e37b3a84",
      "status": "Failed",
      "startTime": "2026-10-11T22:50:53.504973281Z",
      "endTime": "2026-10-11T23:14:57.954117538Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tcruuiers9mtsh6kof4evdrtdk9p3e4 still exists (deletion dispatched at 2026-10-11T22:50:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109405118775431168
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining ...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc79feec4060>: 
    resource group "np-autoscaling-xtg8j7kg4fwz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-xtg8j7kg4fwz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "name": "6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "status": "Failed",
      "startTime": "2026-10-12T02:47:19.509292457Z",
      "endTime": "2026-10-12T03:11:19.252244159Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc79feec4060>: 
    resource group "np-autoscaling-xtg8j7kg4fwz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-xtg8j7kg4fwz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "name": "6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "status": "Failed",
      "startTime": "2026-10-12T02:47:19.509292457Z",
      "endTime": "2026-10-12T03:11:19.252244159Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109448556992532480
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x99270993880>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x99270993880>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107977652647235584
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e12.22%1 of 45 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x2fbb237581e0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kc94rh5nhxb8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kc94rh5nhxb8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2fbb237ab310>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x2fbb237581e0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-kc94rh5nhxb8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-kc94rh5nhxb8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2fbb237ab310>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-kc94rh5nhxb8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107763785899970560

Environment: STG

Failure patterns: 14 · Failures matched: 112 · Total runs: 20 · Runs affected: 12

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e735.00%7 of 20 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 6: 18 · Oct 7: 40 · Oct 8: 9 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x13cc8f58b420>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sz5gdnfzqhdd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda",
      "name": "2c72bb6d-878b-457e-bf39-56f98851abda",
      "status": "Failed",
      "startTime": "2026-10-09T06:39:07.400427011Z",
      "endTime": "2026-10-09T07:08:09.774067703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x13cc8f58b420>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sz5gdnfzqhdd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda",
      "name": "2c72bb6d-878b-457e-bf39-56f98851abda",
      "status": "Failed",
      "startTime": "2026-10-09T06:39:07.400427011Z",
      "endTime": "2026-10-09T07:08:09.774067703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3c5e53f64c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p8759llzs4pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "name": "ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:57.884545997Z",
      "endTime": "2026-10-08T01:47:57.229221309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n5sb8-kxqvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n5sb8-kxqvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3c5e53f64c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p8759llzs4pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "name": "ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:57.884545997Z",
      "endTime": "2026-10-08T01:47:57.229221309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n5sb8-kxqvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n5sb8-kxqvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm"
Unexpected error:
    <*fmt.wrapError | 0x25efd83de780>: 
    failed to create NodePool np-drrgfm: failed waiting for nodepool="np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm" in resourcegroup="rg-candidate-5-1-7sfzz5wmdjlq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b",
      "name": "c439fc17-ab56-478e-a949-fbf669f84a7b",
      "status": "Failed",
      "startTime": "2026-10-08T01:47:33.608428033Z",
      "endTime": "2026-10-08T02:16:36.341701384Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm"
Unexpected error:
    <*fmt.wrapError | 0x25efd83de780>: 
    failed to create NodePool np-drrgfm: failed waiting for nodepool="np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm" in resourcegroup="rg-candidate-5-1-7sfzz5wmdjlq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b",
      "name": "c439fc17-ab56-478e-a949-fbf669f84a7b",
      "status": "Failed",
      "startTime": "2026-10-08T01:47:33.608428033Z",
      "endTime": "2026-10-08T02:16:36.341701384Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (25)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.204
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.214
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.224
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.234
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.04
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.14
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2108427018348531712
n/a2107985213920382976
n/a2107969374252961792
n/a2107827709018116096
n/a2107763728446394368
n/a2107701758112305152
n/a2107652362595209216
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e735.00%7 of 20 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsOct 6: 0 · Oct 7: 5 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1 · Oct 12: 0 · Oct 13: 0DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1003aae08080>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dvn947jdn4hc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c051cc9-3f87-43a3-86ce-d9c6d0efccb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c051cc9-3f87-43a3-86ce-d9c6d0efccb9",
      "name": "3c051cc9-3f87-43a3-86ce-d9c6d0efccb9",
      "status": "Failed",
      "startTime": "2026-10-11T23:48:35.970441715Z",
      "endTime": "2026-10-12T00:17:41.715765376Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dvn947jdn4hc--managed/providers/Microsoft.Compute/virtualMachines/q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dvn947jdn4hc--managed/providers/Microsoft.Compute/virtualMachines/q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c051cc9-3f87-43a3-86ce-d9c6d0efccb9
            correlation ID: e2102b28-bcd5-415f-bde1-1dd34ba167b4
            client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x1003aae08080>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dvn947jdn4hc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c051cc9-3f87-43a3-86ce-d9c6d0efccb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c051cc9-3f87-43a3-86ce-d9c6d0efccb9",
      "name": "3c051cc9-3f87-43a3-86ce-d9c6d0efccb9",
      "status": "Failed",
      "startTime": "2026-10-11T23:48:35.970441715Z",
      "endTime": "2026-10-12T00:17:41.715765376Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dvn947jdn4hc--managed/providers/Microsoft.Compute/virtualMachines/q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dvn947jdn4hc--managed/providers/Microsoft.Compute/virtualMachines/q7p0f9j6y7m4m7c-np-deletable-glcrn-8z7zf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c051cc9-3f87-43a3-86ce-d9c6d0efccb9
            correlation ID: e2102b28-bcd5-415f-bde1-1dd34ba167b4
            client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xcfd18f3b720>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r4vhjgdwsnjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "name": "d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "status": "Failed",
      "startTime": "2026-10-08T02:22:41.727850228Z",
      "endTime": "2026-10-08T02:51:47.319042789Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xcfd18f3b720>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r4vhjgdwsnjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "name": "d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "status": "Failed",
      "startTime": "2026-10-08T02:22:41.727850228Z",
      "endTime": "2026-10-08T02:51:47.319042789Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x115d2dac76c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-knjdfr9bnmx4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "name": "3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "status": "Failed",
      "startTime": "2026-10-07T23:45:51.426276166Z",
      "endTime": "2026-10-08T00:14:52.690397112Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x115d2dac76c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-knjdfr9bnmx4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "name": "3365f6a5-3dc7-4c04-bf39-d10dc605849e",
      "status": "Failed",
      "startTime": "2026-10-07T23:45:51.426276166Z",
      "endTime": "2026-10-08T00:14:52.690397112Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-vn2nz to exist\nMachine u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-knjdfr9bnmx4--managed/providers/Microsoft.Compute/virtualMachines/u0h1u4p4y0s3z3s-np-deletable-2zzc7-xpmzt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2109418788515483648
n/a2107985213920382976
n/a2107969374252961792
n/a2107827709018116096
n/a2107763728446394368
n/a2107701758112305152
n/a2107652362595209216
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
e2e630.00%6 of 20 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsOct 6: 5 · Oct 7: 5 · Oct 8: 5 · Oct 9: 5 · Oct 10: 5 · Oct 11: 5 · Oct 12: 4 · Oct 13: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x26308605f950>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vb66qj77mjrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vb66qj77mjrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: cc37dc3c-22b2-4fec-a5d0-2aba23ea3ad5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x26308605f950>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vb66qj77mjrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vb66qj77mjrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: cc37dc3c-22b2-4fec-a5d0-2aba23ea3ad5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xb11fa465920>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vsn9q285vfrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vsn9q285vfrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 264d5240-5762-4b86-9192-64fd260f49fd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xb11fa465920>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vsn9q285vfrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-vsn9q285vfrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 264d5240-5762-4b86-9192-64fd260f49fd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3ab272e5a090>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-97t8mc9zr7vt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-97t8mc9zr7vt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: b25da80c-bc0c-47f5-b073-ca14b3b999a7
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3ab272e5a090>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-97t8mc9zr7vt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-97t8mc9zr7vt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: b25da80c-bc0c-47f5-b073-ca14b3b999a7
            client request ID: unavailable
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors5
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2109781178767642624
n/a2109418788515483648
n/a2109056606967697408
n/a2108694214664523776
n/a2108331766338031616
n/a2107969374252961792
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e630.00%6 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 8 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x273477d04048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zvv7lmwhhzgh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "name": "81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "status": "Failed",
      "startTime": "2026-10-08T01:30:01.277978624Z",
      "endTime": "2026-10-08T01:59:06.873419943Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x273477d04048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zvv7lmwhhzgh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "name": "81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "status": "Failed",
      "startTime": "2026-10-08T01:30:01.277978624Z",
      "endTime": "2026-10-08T01:59:06.873419943Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x26a27898a0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-m9xrgvjx8lwn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "name": "e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "status": "Failed",
      "startTime": "2026-10-07T23:17:33.092346932Z",
      "endTime": "2026-10-07T23:46:38.337633525Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-whw2v-lppqg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-whw2v-lppqg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x26a27898a0d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-m9xrgvjx8lwn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "name": "e7fd4c79-ee4a-4813-89bb-a9a38553de85",
      "status": "Failed",
      "startTime": "2026-10-07T23:17:33.092346932Z",
      "endTime": "2026-10-07T23:46:38.337633525Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-whw2v-lppqg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-whw2v-lppqg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-lppqg to exist\nMachine cilium-cluster-cilium-np-whw2v-x7t8r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-m9xrgvjx8lwn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-whw2v-x7t8r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x390344d20c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4wvdmxwppl4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80",
      "name": "02042050-d676-4d90-bee1-de9c86994c80",
      "status": "Failed",
      "startTime": "2026-10-07T14:30:29.815037524Z",
      "endTime": "2026-10-07T14:59:36.125232922Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x390344d20c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4wvdmxwppl4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/02042050-d676-4d90-bee1-de9c86994c80",
      "name": "02042050-d676-4d90-bee1-de9c86994c80",
      "status": "Failed",
      "startTime": "2026-10-07T14:30:29.815037524Z",
      "endTime": "2026-10-07T14:59:36.125232922Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-56qxc-gqmsv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-gqmsv to exist\nMachine cilium-cluster-cilium-np-56qxc-sft5q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4wvdmxwppl4j--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-56qxc-sft5q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2107985213920382976
n/a2107969374252961792
n/a2107827709018116096
n/a2107763728446394368
n/a2107701758112305152
n/a2107652362595209216
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e210.00%2 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 2 · Oct 12: 5 · Oct 13: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-mjltzvcvvmbh/cluster-nightly-4-22-xv7w42 should provision
Unexpected error:
    <*fmt.wrapError | 0xe85a0514ac0>: 
    failed to create HCP cluster cluster-nightly-4-22-xv7w42: failed waiting for cluster="cluster-nightly-4-22-xv7w42" in resourcegroup="rg-nightly-4-22-mjltzvcvvmbh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df7b36b8-c15c-41a8-87f0-fb3ee1d8a874
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df7b36b8-c15c-41a8-87f0-fb3ee1d8a874",
      "name": "df7b36b8-c15c-41a8-87f0-fb3ee1d8a874",
      "status": "Failed",
      "startTime": "2026-10-12T23:26:58.898405913Z",
      "endTime": "2026-10-12T23:56:08.010677982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df7b36b8-c15c-41a8-87f0-fb3ee1d8a874
            correlation ID: 99498835-8536-458a-9168-4524819cc6bd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-mjltzvcvvmbh/cluster-nightly-4-22-xv7w42 should provision
Unexpected error:
    <*fmt.wrapError | 0xe85a0514ac0>: 
    failed to create HCP cluster cluster-nightly-4-22-xv7w42: failed waiting for cluster="cluster-nightly-4-22-xv7w42" in resourcegroup="rg-nightly-4-22-mjltzvcvvmbh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df7b36b8-c15c-41a8-87f0-fb3ee1d8a874
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df7b36b8-c15c-41a8-87f0-fb3ee1d8a874",
      "name": "df7b36b8-c15c-41a8-87f0-fb3ee1d8a874",
      "status": "Failed",
      "startTime": "2026-10-12T23:26:58.898405913Z",
      "endTime": "2026-10-12T23:56:08.010677982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df7b36b8-c15c-41a8-87f0-fb3ee1d8a874
            correlation ID: 99498835-8536-458a-9168-4524819cc6bd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-lmbgqw" at install version 4.22.0-0.nightly-multi-2026-10-12-042614
Unexpected error:
    <*fmt.wrapError | 0x2912ce16d00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-lmbgqw: failed waiting for cluster="cp-ystream-upgrade-4-23-lmbgqw" in resourcegroup="rg-cp-ystream-upgrade-4-23-xm2k5t9rnhbb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40887f6f-9dfa-4fd4-a671-e1259364b51d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40887f6f-9dfa-4fd4-a671-e1259364b51d",
      "name": "40887f6f-9dfa-4fd4-a671-e1259364b51d",
      "status": "Failed",
      "startTime": "2026-10-12T23:30:45.035752487Z",
      "endTime": "2026-10-12T23:59:45.878026173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40887f6f-9dfa-4fd4-a671-e1259364b51d
            correlation ID: 61a61582-38bf-4dc1-a572-663613c43a27
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-lmbgqw" at install version 4.22.0-0.nightly-multi-2026-10-12-042614
Unexpected error:
    <*fmt.wrapError | 0x2912ce16d00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-lmbgqw: failed waiting for cluster="cp-ystream-upgrade-4-23-lmbgqw" in resourcegroup="rg-cp-ystream-upgrade-4-23-xm2k5t9rnhbb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40887f6f-9dfa-4fd4-a671-e1259364b51d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40887f6f-9dfa-4fd4-a671-e1259364b51d",
      "name": "40887f6f-9dfa-4fd4-a671-e1259364b51d",
      "status": "Failed",
      "startTime": "2026-10-12T23:30:45.035752487Z",
      "endTime": "2026-10-12T23:59:45.878026173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/40887f6f-9dfa-4fd4-a671-e1259364b51d
            correlation ID: 61a61582-38bf-4dc1-a572-663613c43a27
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-gkft6x" at install version 4.20.0-0.nightly-multi-2026-10-11-194900
Unexpected error:
    <*fmt.wrapError | 0x168e93083220>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-gkft6x: failed waiting for cluster="cp-ystream-upgrade-4-21-gkft6x" in resourcegroup="rg-cp-ystream-upgrade-4-21-p9dzhljjxbzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/87c1b701-975b-48a5-946a-b69c35657d88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/87c1b701-975b-48a5-946a-b69c35657d88",
      "name": "87c1b701-975b-48a5-946a-b69c35657d88",
      "status": "Failed",
      "startTime": "2026-10-12T23:28:15.763925029Z",
      "endTime": "2026-10-12T23:57:15.727344969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/87c1b701-975b-48a5-946a-b69c35657d88
            correlation ID: 15d5486b-2716-4d22-b4be-42c25e5289af
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-gkft6x" at install version 4.20.0-0.nightly-multi-2026-10-11-194900
Unexpected error:
    <*fmt.wrapError | 0x168e93083220>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-gkft6x: failed waiting for cluster="cp-ystream-upgrade-4-21-gkft6x" in resourcegroup="rg-cp-ystream-upgrade-4-21-p9dzhljjxbzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/87c1b701-975b-48a5-946a-b69c35657d88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/87c1b701-975b-48a5-946a-b69c35657d88",
      "name": "87c1b701-975b-48a5-946a-b69c35657d88",
      "status": "Failed",
      "startTime": "2026-10-12T23:28:15.763925029Z",
      "endTime": "2026-10-12T23:57:15.727344969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/87c1b701-975b-48a5-946a-b69c35657d88
            correlation ID: 15d5486b-2716-4d22-b4be-42c25e5289af
            client request ID: unavailable
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2109781178767642624
n/a2109418788515483648
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e210.00%2 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1 · Oct 12: 1 · Oct 13: 0PROD
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
    <*fmt.wrapError | 0x2d73d9d10e00>: 
    failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
      "name": "41bf70ba-7344-4c62-9364-b6165697473d",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:02.793421826Z",
      "endTime": "2026-10-12T23:56:08.063137348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
            correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
    <*fmt.wrapError | 0x2d73d9d10e00>: 
    failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
      "name": "41bf70ba-7344-4c62-9364-b6165697473d",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:02.793421826Z",
      "endTime": "2026-10-12T23:56:08.063137348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
            correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-phmgklt4jbx8/cluster-nightly-5-0-dsvpsx should provision
Unexpected error:
    <*fmt.wrapError | 0x174e773b4b40>: 
    failed to create HCP cluster cluster-nightly-5-0-dsvpsx: failed waiting for cluster="cluster-nightly-5-0-dsvpsx" in resourcegroup="rg-nightly-5-0-phmgklt4jbx8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1dcda58e-002a-4f82-9591-34d179e38dd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1dcda58e-002a-4f82-9591-34d179e38dd7",
      "name": "1dcda58e-002a-4f82-9591-34d179e38dd7",
      "status": "Failed",
      "startTime": "2026-10-11T23:24:10.263042602Z",
      "endTime": "2026-10-11T23:53:11.680283328Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, azure-cloud-controller-manager, catalog-operator, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, openshift-controller-manager, cluster-policy-controller, kube-controller-manager, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, konnectivity-agent, cluster-image-registry-operator, cluster-version-operator, ingress-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1dcda58e-002a-4f82-9591-34d179e38dd7
            correlation ID: 8da2e5e4-c1bb-4598-8408-0edb1c690ef8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-phmgklt4jbx8/cluster-nightly-5-0-dsvpsx should provision
Unexpected error:
    <*fmt.wrapError | 0x174e773b4b40>: 
    failed to create HCP cluster cluster-nightly-5-0-dsvpsx: failed waiting for cluster="cluster-nightly-5-0-dsvpsx" in resourcegroup="rg-nightly-5-0-phmgklt4jbx8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1dcda58e-002a-4f82-9591-34d179e38dd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1dcda58e-002a-4f82-9591-34d179e38dd7",
      "name": "1dcda58e-002a-4f82-9591-34d179e38dd7",
      "status": "Failed",
      "startTime": "2026-10-11T23:24:10.263042602Z",
      "endTime": "2026-10-11T23:53:11.680283328Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, azure-cloud-controller-manager, catalog-operator, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, openshift-controller-manager, cluster-policy-controller, kube-controller-manager, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, konnectivity-agent, cluster-image-registry-operator, cluster-version-operator, ingress-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1dcda58e-002a-4f82-9591-34d179e38dd7
            correlation ID: 8da2e5e4-c1bb-4598-8408-0edb1c690ef8
            client request ID: unavailable
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2109781178767642624
n/a2109418788515483648
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining ...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 6: 6 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x86f41c2a240>: 
    resource group "rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "name": "1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:56.823039116Z",
      "endTime": "2026-10-08T01:42:56.76971285Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x86f41c2a240>: 
    resource group "rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "name": "1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:56.823039116Z",
      "endTime": "2026-10-08T01:42:56.76971285Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107985213920382976
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
    <*fmt.wrapError | 0xc4ca110a7e0>: 
    failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:12.849511271Z",
      "endTime": "2026-10-12T23:56:15.713757928Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
            correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
    <*fmt.wrapError | 0xc4ca110a7e0>: 
    failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
      "status": "Failed",
      "startTime": "2026-10-12T23:27:12.849511271Z",
      "endTime": "2026-10-12T23:56:15.713757928Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
            correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109781178767642624
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1 · Oct 12: 0 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-nb9jxpzlfmcp/cluster-nightly-5-1-wkbflh should provision
Unexpected error:
    <*fmt.wrapError | 0x1d0dfed5cd40>: 
    failed to create HCP cluster cluster-nightly-5-1-wkbflh: failed waiting for cluster="cluster-nightly-5-1-wkbflh" in resourcegroup="rg-nightly-5-1-nb9jxpzlfmcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4b00e07-8ca7-4a34-a5ca-40ac469ad96a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4b00e07-8ca7-4a34-a5ca-40ac469ad96a",
      "name": "e4b00e07-8ca7-4a34-a5ca-40ac469ad96a",
      "status": "Failed",
      "startTime": "2026-10-11T23:24:11.419225738Z",
      "endTime": "2026-10-11T23:53:12.203667325Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-181926 is Partial (want Completed), started 22m49s ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4b00e07-8ca7-4a34-a5ca-40ac469ad96a
            correlation ID: b50e9c9e-d4f4-4ba1-9e2c-863be02b98b4
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-nb9jxpzlfmcp/cluster-nightly-5-1-wkbflh should provision
Unexpected error:
    <*fmt.wrapError | 0x1d0dfed5cd40>: 
    failed to create HCP cluster cluster-nightly-5-1-wkbflh: failed waiting for cluster="cluster-nightly-5-1-wkbflh" in resourcegroup="rg-nightly-5-1-nb9jxpzlfmcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4b00e07-8ca7-4a34-a5ca-40ac469ad96a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4b00e07-8ca7-4a34-a5ca-40ac469ad96a",
      "name": "e4b00e07-8ca7-4a34-a5ca-40ac469ad96a",
      "status": "Failed",
      "startTime": "2026-10-11T23:24:11.419225738Z",
      "endTime": "2026-10-11T23:53:12.203667325Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-181926 is Partial (want Completed), started 22m49s ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4b00e07-8ca7-4a34-a5ca-40ac469ad96a
            correlation ID: b50e9c9e-d4f4-4ba1-9e2c-863be02b98b4
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109418788515483648
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x371cbc791620>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
      "name": "7da88189-d1ef-4299-b869-95f7299be069",
      "status": "Failed",
      "startTime": "2026-10-12T23:47:30.718672913Z",
      "endTime": "2026-10-13T00:16:35.706367535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
            correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x371cbc791620>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
      "name": "7da88189-d1ef-4299-b869-95f7299be069",
      "status": "Failed",
      "startTime": "2026-10-12T23:47:30.718672913Z",
      "endTime": "2026-10-13T00:16:35.706367535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
            correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109781178767642624
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down cluster resources, managed cluster, hosted cluster, namespaces; [descendantResources] remaining resources: applydesires, externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27901568a690>: 
    resource group "external-auth-cluster-ch86qj77dnn4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-ch86qj77dnn4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87",
      "name": "0cf8be7c-97b3-497d-b733-73f64068db87",
      "status": "Failed",
      "startTime": "2026-10-07T06:22:44.810261014Z",
      "endTime": "2026-10-07T06:46:52.177879454Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0x279015820520>{
                msg: "resource group \"external-auth-cluster-ch86qj77dnn4\": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0x279015820460>{
                    msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*fmt.wrapError | 0x279015820440>{
                        msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                        err: <*fmt.wrapError | 0x279015820420>{
                            msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            err: <*exported.ResponseError | 0x27901606a960>{
                                ErrorCode: "InternalServerError",
                                StatusCode: 200,
                                RawResponse: {
                                    Status: "200 OK",
                                    StatusCode: 200,
                                    Proto: "HTTP/2.0",
                                    ProtoMajor: 2,
                                    ProtoMinor: 0,
                                    Header: {
                                        "Cache-Control": [...],
                                        "X-Envoy-Upstream-Service-Time": [...],
                                        "X-Ms-Operation-Identifier": [...],
                                        "X-Ms-Correlation-Request-Id": [...],
                                        "X-Msedge-Ref": [...],
                                        "Pragma": [...],
                                        "Content-Type": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": [...],
                                        "Strict-Transport-Security": [...],
                                        "X-Content-Type-Options": [...],
                                        "Date": [...],
                                        "Content-Length": [...],
                                        "X-Ms-Request-Id": [...],
                                        "X-Ms-Routing-Request-Id": [...],
                                        "X-Cache": [...],
                                        "Expires": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": [...],
                                    },
                                    Body: <*exported.nopClosingBytesReader | 0x279015820220>{
                                        s: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        i: 0,
                                    },
                                    ContentLength: 1273,
                                    TransferEncoding: nil,
                                    Close: false,
                                    Uncompressed: false,
                                    Trailer: nil,
                                    Request: {
                                        Method: "GET",
                                        URL: {Scheme: ..., Opaque: ..., User: ..., Host: ..., Path: ..., Fragment: ..., RawQuery: ..., RawPath: ..., RawFragment: ..., ForceQuery: ..., OmitHost: ...},
                                        Proto: "HTTP/1.1",
                                        ProtoMajor: 1,
                                        ProtoMinor: 1,
                                        Header: {...: ..., ...: ...},
                                        Body: nil,
                                        GetBody: nil,
                                        ContentLength: 0,
                                        TransferEncoding: nil,
                                        Close: false,
                                        Host: "management.azure.com",
                                        Form: nil,
                                        PostForm: nil,
                                        MultipartForm: nil,
                                        Trailer: nil,
                                        RemoteAddr: "",
                                        RequestURI: "",
                                        TLS: nil,
                                        Cancel: nil,
                                        Response: nil,
                                        Pattern: "",
                                        ctx: <suppressed context>,
                                        pat: nil,
                                        matches: nil,
                                        otherValues: nil,
                                    },
                                    TLS: {
                                        Version: 772,
                                        HandshakeComplete: true,
                                        DidResume: false,
                                        CipherSuite: 4866,
                                        CurveID: 25,
                                        NegotiatedProtocol: "h2",
                                        NegotiatedProtocolIsMutual: true,
                                        ServerName: "management.azure.com",
                                        PeerCertificates: [..., ..., ...],
                                        VerifiedChains: [...],
                                        SignedCertificateTimestamps: nil,
                                        OCSPResponse: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        TLSUnique: nil,
                                        ECHAccepted: false,
                                        HelloRetryRequest: true,
                                        ekm: 0x585e900,
                                        testingOnlyPeerSignatureAlgorithm: 2052,
                                    },
                                },
                                errMsg: "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x27901568a690>: 
    resource group "external-auth-cluster-ch86qj77dnn4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-ch86qj77dnn4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87",
      "name": "0cf8be7c-97b3-497d-b733-73f64068db87",
      "status": "Failed",
      "startTime": "2026-10-07T06:22:44.810261014Z",
      "endTime": "2026-10-07T06:46:52.177879454Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0x279015820520>{
                msg: "resource group \"external-auth-cluster-ch86qj77dnn4\": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0x279015820460>{
                    msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*fmt.wrapError | 0x279015820440>{
                        msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                        err: <*fmt.wrapError | 0x279015820420>{
                            msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch86qj77dnn4\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            err: <*exported.ResponseError | 0x27901606a960>{
                                ErrorCode: "InternalServerError",
                                StatusCode: 200,
                                RawResponse: {
                                    Status: "200 OK",
                                    StatusCode: 200,
                                    Proto: "HTTP/2.0",
                                    ProtoMajor: 2,
                                    ProtoMinor: 0,
                                    Header: {
                                        "Cache-Control": [...],
                                        "X-Envoy-Upstream-Service-Time": [...],
                                        "X-Ms-Operation-Identifier": [...],
                                        "X-Ms-Correlation-Request-Id": [...],
                                        "X-Msedge-Ref": [...],
                                        "Pragma": [...],
                                        "Content-Type": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": [...],
                                        "Strict-Transport-Security": [...],
                                        "X-Content-Type-Options": [...],
                                        "Date": [...],
                                        "Content-Length": [...],
                                        "X-Ms-Request-Id": [...],
                                        "X-Ms-Routing-Request-Id": [...],
                                        "X-Cache": [...],
                                        "Expires": [...],
                                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": [...],
                                    },
                                    Body: <*exported.nopClosingBytesReader | 0x279015820220>{
                                        s: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        i: 0,
                                    },
                                    ContentLength: 1273,
                                    TransferEncoding: nil,
                                    Close: false,
                                    Uncompressed: false,
                                    Trailer: nil,
                                    Request: {
                                        Method: "GET",
                                        URL: {Scheme: ..., Opaque: ..., User: ..., Host: ..., Path: ..., Fragment: ..., RawQuery: ..., RawPath: ..., RawFragment: ..., ForceQuery: ..., OmitHost: ...},
                                        Proto: "HTTP/1.1",
                                        ProtoMajor: 1,
                                        ProtoMinor: 1,
                                        Header: {...: ..., ...: ...},
                                        Body: nil,
                                        GetBody: nil,
                                        ContentLength: 0,
                                        TransferEncoding: nil,
                                        Close: false,
                                        Host: "management.azure.com",
                                        Form: nil,
                                        PostForm: nil,
                                        MultipartForm: nil,
                                        Trailer: nil,
                                        RemoteAddr: "",
                                        RequestURI: "",
                                        TLS: nil,
                                        Cancel: nil,
                                        Response: nil,
                                        Pattern: "",
                                        ctx: <suppressed context>,
                                        pat: nil,
                                        matches: nil,
                                        otherValues: nil,
                                    },
                                    TLS: {
                                        Version: 772,
                                        HandshakeComplete: true,
                                        DidResume: false,
                                        CipherSuite: 4866,
                                        CurveID: 25,
                                        NegotiatedProtocol: "h2",
                                        NegotiatedProtocolIsMutual: true,
                                        ServerName: "management.azure.com",
                                        PeerCertificates: [..., ..., ...],
                                        VerifiedChains: [...],
                                        SignedCertificateTimestamps: nil,
                                        OCSPResponse: [..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...],
                                        TLSUnique: nil,
                                        ECHAccepted: false,
                                        HelloRetryRequest: true,
                                        ekm: 0x585e900,
                                        testingOnlyPeerSignatureAlgorithm: 2052,
                                    },
                                },
                                errMsg: "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"name\": \"0cf8be7c-97b3-497d-b733-73f64068db87\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-10-07T06:22:44.810261014Z\",\n  \"endTime\": \"2026-10-07T06:46:52.177879454Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t9p58rkrigg6n9uc8454ajabpln3t9n still exists (deletion dispatched at 2026-10-07T06:22:45Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 6 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down cluster resources (3), managed cluster (1), hosted cluster (1), namespaces (2)\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107701758112305152
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
full failure pattern:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x3d800eb17200>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3d800f297260>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3d800f297240>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e191e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3d800e204000>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3d800eb05ee0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e0fab40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x3d800eb17200>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3d800f297260>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3d800f297240>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e191e00>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3d800e204000>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3d800eb05ee0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3d800e0fab40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2109781178767642624
VerifyBasicAccess failed: failed to list services: Unauthorized
full failure pattern:
VerifyBasicAccess failed: failed to list services: Unauthorized
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-20-g5bgdz" is viable
Unexpected error:
    <*errors.joinError | 0x2608a89861e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2608a90941e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2608a90941c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2608a81192c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2608a86f4180>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2608a86f4100>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2608a7f286e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-20-g5bgdz" is viable
Unexpected error:
    <*errors.joinError | 0x2608a89861e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2608a90941e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2608a90941c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2608a81192c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2608a86f4180>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2608a86f4100>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2608a7f286e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108331766338031616
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>
full failure pattern:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e15.00%1 of 20 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x2655580ad6c0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x2655580ad6c0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:55923->172.30.0.10:53: read: connection refused",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2107827709018116096