CIHealth Failure Patterns

Environment: DEV

Failure patterns: 59 · Failures matched: 143 · Total runs: 92 · Runs affected: 68

Failure pattern

Failed at

Trend

Also in

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x22128cbeb10>: 
    GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "name": "26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:37.259328289Z",
      "endTime": "2026-10-19T11:01:44.306693815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-node-tuning-operator, certified-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, dns-operator, ingress-operator, redhat-operators-catalog, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
            correlation ID: 87b43167-8612-41cf-b2af-01e82520620b
            client request ID: 826cfb23-3364-490c-b9f1-549ca623afb4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x22128cbeb10>: 
    GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "name": "26184216-4ab9-48f4-b1f2-ce27f41ac9be",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:37.259328289Z",
      "endTime": "2026-10-19T11:01:44.306693815Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-node-tuning-operator, certified-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, dns-operator, ingress-operator, redhat-operators-catalog, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26184216-4ab9-48f4-b1f2-ce27f41ac9be
            correlation ID: 87b43167-8612-41cf-b2af-01e82520620b
            client request ID: 826cfb23-3364-490c-b9f1-549ca623afb4
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x22a4ab41e440>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-tkvqmscnx8hd" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25",
      "name": "7966e350-c6be-4e91-9c2e-1655783cec25",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:11.987681126Z",
      "endTime": "2026-10-19T11:01:14.305309512Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
            correlation ID: 8f412970-2932-49aa-85c1-e501fdb88128
            client request ID: 037bff49-0293-4f3d-ab7f-9ff8b71e3d4a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x22a4ab41e440>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-tkvqmscnx8hd" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25",
      "name": "7966e350-c6be-4e91-9c2e-1655783cec25",
      "status": "Failed",
      "startTime": "2026-10-19T10:32:11.987681126Z",
      "endTime": "2026-10-19T11:01:14.305309512Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966e350-c6be-4e91-9c2e-1655783cec25
            correlation ID: 8f412970-2932-49aa-85c1-e501fdb88128
            client request ID: 037bff49-0293-4f3d-ab7f-9ff8b71e3d4a
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 09:43	PR #7582	2112117383177965568

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after ad... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x2ba9acdcc318>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-pp8lljnf5t7j" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "name": "025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "status": "Failed",
      "startTime": "2026-10-19T16:45:45.996504863Z",
      "endTime": "2026-10-19T17:04:48.768522414Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-7v7tf-6wgpp: Failed: availabilitysets failed to create or update. err: failed to get existing resource cni-cilium-pp8lljnf5t7j--managed/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-pp8lljnf5t7j--managed/providers/Microsoft.Compute/availabilitySets/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after ad... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
            correlation ID: c38c42ba-d929-4393-9a38-4d460e6cc1c1
            client request ID: 991ecdc6-7a24-4e68-a023-12a15de7fc26
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x2ba9acdcc318>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-pp8lljnf5t7j" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "name": "025c0379-fa3b-4e93-b19d-0ad03edccdd2",
      "status": "Failed",
      "startTime": "2026-10-19T16:45:45.996504863Z",
      "endTime": "2026-10-19T17:04:48.768522414Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-7v7tf-6wgpp: Failed: availabilitysets failed to create or update. err: failed to get existing resource cni-cilium-pp8lljnf5t7j--managed/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-pp8lljnf5t7j--managed/providers/Microsoft.Compute/availabilitySets/2thvfqe1vdauvp1vi8ns5oibmpnkopfg_cilium-cl-cilium-np-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after ad... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/025c0379-fa3b-4e93-b19d-0ad03edccdd2
            correlation ID: c38c42ba-d929-4393-9a38-4d460e6cc1c1
            client request ID: 991ecdc6-7a24-4e68-a023-12a15de7fc26
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 15:37	PR #6920	2112206455762325504

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are ... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p"
Unexpected error:
    <*fmt.wrapError | 0x2d7273ae000>: 
    failed to create NodePool np-vkgl4p: failed waiting for nodepool="np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p" in resourcegroup="rg-hs-pre--j799lrwqxz6j" to finish creating: GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "name": "c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "status": "Failed",
      "startTime": "2026-10-19T22:21:51.863828648Z",
      "endTime": "2026-10-19T22:40:54.499105259Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: Failed: virtualmachine failed to create or update. err: failed to get existing resource rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
            correlation ID: 2e542bba-f131-4b5d-bf4c-a77a8bc95e2d
            client request ID: 83a4b938-e107-403c-83e6-0592ebdaeff2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p"
Unexpected error:
    <*fmt.wrapError | 0x2d7273ae000>: 
    failed to create NodePool np-vkgl4p: failed waiting for nodepool="np-vkgl4p" for cluster "cluster-hs-pre-vkgl4p" in resourcegroup="rg-hs-pre--j799lrwqxz6j" to finish creating: GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "name": "c5e9078f-f162-40b9-b342-a3ebc6cb8256",
      "status": "Failed",
      "startTime": "2026-10-19T22:21:51.863828648Z",
      "endTime": "2026-10-19T22:40:54.499105259Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98: Failed: virtualmachine failed to create or update. err: failed to get existing resource rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--j799lrwqxz6j-vkgl4p--managed/providers/Microsoft.Compute/virtualMachines/f2u7w5o4k6n1j9f-np-vkgl4p-qlncl-xgc98\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1899008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c5e9078f-f162-40b9-b342-a3ebc6cb8256
            correlation ID: 2e542bba-f131-4b5d-bf4c-a77a8bc95e2d
            client request ID: 83a4b938-e107-403c-83e6-0592ebdaeff2
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:10	PR #7597	2112290330521899008

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, openshift-apiserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x169a1fa14360>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-pvwnm5h2lwhh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "name": "201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:39.240419381Z",
      "endTime": "2026-10-19T11:00:44.480106321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, cluster-node-tuning-operator, redhat-marketplace-catalog, catalog-operator, cluster-network-operator, certified-operators-catalog, olm-collect-profiles, openshift-controller-manager, packageserver, machine-approver, cluster-storage-operator, cluster-version-operator, cluster-policy-controller, openshift-route-controller-manager, community-operators-catalog, cluster-image-registry-operator, dns-operator, konnectivity-agent, olm-operator, azure-cloud-controller-manager, csi-snapshot-controller-operator, hosted-cluster-config-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
            correlation ID: a21d9aa7-da4e-4f79-9da1-353faa5bb8d0
            client request ID: 457b6d19-40ff-4036-a44f-729e35535125
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x169a1fa14360>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-pvwnm5h2lwhh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "name": "201f68b6-906f-4c7f-9b20-18d6e82fa913",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:39.240419381Z",
      "endTime": "2026-10-19T11:00:44.480106321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, cluster-node-tuning-operator, redhat-marketplace-catalog, catalog-operator, cluster-network-operator, certified-operators-catalog, olm-collect-profiles, openshift-controller-manager, packageserver, machine-approver, cluster-storage-operator, cluster-version-operator, cluster-policy-controller, openshift-route-controller-manager, community-operators-catalog, cluster-image-registry-operator, dns-operator, konnectivity-agent, olm-operator, azure-cloud-controller-manager, csi-snapshot-controller-operator, hosted-cluster-config-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/201f68b6-906f-4c7f-9b20-18d6e82fa913
            correlation ID: a21d9aa7-da4e-4f79-9da1-353faa5bb8d0
            client request ID: 457b6d19-40ff-4036-a44f-729e35535125
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 09:43	PR #7582	2112117383177965568

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4jxfrx" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3b2f64940d00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4jxfrx: failed waiting for cluster="cp-ystream-upgrade-4-21-4jxfrx" in resourcegroup="rg-cp-ystream-upgrade-4-21-dxmr9r2kr8sj" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "name": "e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:38.502730447Z",
      "endTime": "2026-10-19T11:00:44.46660972Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 23m51s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
            correlation ID: 72780af9-37e8-42d0-b405-3ed5aea87756
            client request ID: 04371127-7fbb-474b-a1aa-c21583ae4e63
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4jxfrx" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3b2f64940d00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4jxfrx: failed waiting for cluster="cp-ystream-upgrade-4-21-4jxfrx" in resourcegroup="rg-cp-ystream-upgrade-4-21-dxmr9r2kr8sj" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "name": "e769daae-d00d-49ae-a5a9-a8aa6f824f09",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:38.502730447Z",
      "endTime": "2026-10-19T11:00:44.46660972Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 23m51s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e769daae-d00d-49ae-a5a9-a8aa6f824f09
            correlation ID: 72780af9-37e8-42d0-b405-3ed5aea87756
            client request ID: 04371127-7fbb-474b-a1aa-c21583ae4e63
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 09:43	PR #7582	2112117383177965568

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x28db69b98ae0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-f6gwn5qzckpq" to finish creating: GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "name": "55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "status": "Failed",
      "startTime": "2026-10-19T19:48:10.999588854Z",
      "endTime": "2026-10-19T20:07:15.605638501Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-wmvjh-tdsrw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-wmvjh-tdsrw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
            correlation ID: 43d991c8-f99d-4e5d-8930-6d417ef255cd
            client request ID: 24277a58-4912-4429-b89c-c6baeb4435fc
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x28db69b98ae0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-f6gwn5qzckpq" to finish creating: GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "name": "55b2a8bb-3d6c-4170-81ce-f8813840550c",
      "status": "Failed",
      "startTime": "2026-10-19T19:48:10.999588854Z",
      "endTime": "2026-10-19T20:07:15.605638501Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-wmvjh-tdsrw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-wmvjh-tdsrw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-f6gwn5qzckpq--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-wmvjh-tdsrw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8677376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/55b2a8bb-3d6c-4170-81ce-f8813840550c
            correlation ID: 43d991c8-f99d-4e5d-8930-6d417ef255cd
            client request ID: 24277a58-4912-4429-b89c-c6baeb4435fc
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 19:03	PR #7602	2112258228648677376

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3895fbd87c80>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-89sq9p7645tk" to finish creating: GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7",
      "name": "111cac39-15a9-4a55-ab81-242565fc5fd7",
      "status": "Failed",
      "startTime": "2026-10-19T19:38:52.630267131Z",
      "endTime": "2026-10-19T19:57:56.018733569Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
            correlation ID: 439e72b2-9c87-4ffb-ac85-0abf45d7f0d2
            client request ID: 07db40a5-8bb3-4389-ac48-a0125b515b0e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3895fbd87c80>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-89sq9p7645tk" to finish creating: GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7",
      "name": "111cac39-15a9-4a55-ab81-242565fc5fd7",
      "status": "Failed",
      "startTime": "2026-10-19T19:38:52.630267131Z",
      "endTime": "2026-10-19T19:57:56.018733569Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-87vjj to exist\nMachine b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-89sq9p7645tk--managed/providers/Microsoft.Compute/virtualMachines/b4k7t7k1b3t7l6w-autoscale-noaz-d4kmh-zqcrh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3708544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/111cac39-15a9-4a55-ab81-242565fc5fd7
            correlation ID: 439e72b2-9c87-4ffb-ac85-0abf45d7f0d2
            client request ID: 07db40a5-8bb3-4389-ac48-a0125b515b0e
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 18:16	PR #7616	2112246631993708544

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

full failure pattern:

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:180]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x222007964a08>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x222007b84d20>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x2220079649d8>{
                    errs: [
                        <*errors.errorString | 0x222008c33e60>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:180]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x222007964a08>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x222007b84d20>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x2220079649d8>{
                    errs: [
                        <*errors.errorString | 0x222008c33e60>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.19:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:03	PR #7602	2112288570218647552

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-<id>: expected

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x59ce69d41c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x59ce69eea20>{
            s: "dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x59ce69d41c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tjz8vhdkt57n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x59ce69eea20>{
            s: "dev/centralus/ci01-j6725632-mgmt-2/ocm-arohcpci01-2thol4rfhgvi5rmvh2p2p3orcjrvp1s4-g7g2x5z2p7o8m8e: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 07:34	PR #6823	2112085059526725632

alert [hcp] KubeAPIDown fired

full failure pattern:

alert [hcp] KubeAPIDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:52Z
  Ended:       2026-10-19T15:04:51Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:46:52Z
  Ended:       2026-10-19T15:06:52Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeAPIDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

alert [hcp] KubeControllerManagerDown fired

full failure pattern:

alert [hcp] KubeControllerManagerDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:29Z
  Ended:       2026-10-19T15:04:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:47:27Z
  Ended:       2026-10-19T15:07:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeControllerManagerDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

alert [hcp] KubeSchedulerDown fired

full failure pattern:

alert [hcp] KubeSchedulerDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:54Z
  Ended:       2026-10-19T15:06:57Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:46:54Z
  Ended:       2026-10-19T15:04:55Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeSchedulerDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

alert [hcp] KubeletDown fired

full failure pattern:

alert [hcp] KubeletDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T14:46:14Z
  Ended:       2026-10-19T15:04:13Z
  Severity:    Sev2
  Labels:      alertname="KubeletDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: Kubelet has disappeared from Prometheus target discovery.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T14:47:17Z
  Ended:       2026-10-19T15:07:16Z
  Severity:    Sev2
  Labels:      alertname="KubeletDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: Kubelet has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeletDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

alert [svc] KubeAPIDown fired

full failure pattern:

alert [svc] KubeAPIDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:23:41Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-svc", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:41Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:41Z
  Severity:    Sev2
  Labels:      alertname="KubeAPIDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeAPI has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeAPIDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

alert [svc] KubeControllerManagerDown fired

full failure pattern:

alert [svc] KubeControllerManagerDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:23:28Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-svc", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:27Z
  Severity:    Sev2
  Labels:      alertname="KubeControllerManagerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeControllerManager has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeControllerManagerDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

alert [svc] KubeSchedulerDown fired

full failure pattern:

alert [svc] KubeSchedulerDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:23:34Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-svc", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:35Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-1", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-19T14:27:35Z
  Severity:    Sev2
  Labels:      alertname="KubeSchedulerDown", cluster="ci01-j2153856-mgmt-2", component="kubernetes-infrastructure", region="canadacentral", severity="critical"
  Description: KubeScheduler has disappeared from Prometheus target discovery.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeSchedulerDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856

expected <count> ready (and schedulable) nodes, found <count>; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [<node>, <node>]

full failure pattern:

expected <count> ready (and schedulable) nodes, found <count>; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [<node>, <node>]

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:373]: expected 2 ready and schedulable nodes after the cordon/drain/uncordon cycle
Expected success, but got an error:
    <*errors.errorString | 0x3f000f2687f0>: 
    expected 2 ready (and schedulable) nodes in node pool "cordon-np", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]
    {
        s: "expected 2 ready (and schedulable) nodes in node pool \"cordon-np\", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:373]: expected 2 ready and schedulable nodes after the cordon/drain/uncordon cycle
Expected success, but got an error:
    <*errors.errorString | 0x3f000f2687f0>: 
    expected 2 ready (and schedulable) nodes in node pool "cordon-np", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]
    {
        s: "expected 2 ready (and schedulable) nodes in node pool \"cordon-np\", found 0; matching nodes: ; all nodes by pool (2 total): <no-nodepool-label>: [f8d4b9b3r0n2i8f-cordon-np-h6xll-c5xxx, f8d4b9b3r0n2i8f-cordon-np-h6xll-k9728]",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 07:05	PR #7597	2112077587801968640

node pool "<nodepool>" should have at least 2 nodes (autoscaling min), found 0

full failure pattern:

node pool "<nodepool>" should have at least 2 nodes (autoscaling min), found 0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:519]: Timed out after 900.005s.
node labels/taints should appear on the intended node pools of cluster "agg-cluster"
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:506 with:
node pool "agg-np-a" should have at least 2 nodes (autoscaling min), found 0
Expected
    <int>: 0
to be >=
    <int>: 2
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:519]: Timed out after 900.005s.
node labels/taints should appear on the intended node pools of cluster "agg-cluster"
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:506 with:
node pool "agg-np-a" should have at least 2 nodes (autoscaling min), found 0
Expected
    <int>: 0
to be >=
    <int>: 2

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 07:05	PR #7597	2112077587801968640

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:44:19Z
  Ended:       2026-10-19T22:50:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4417280-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti4iojordd6hm0nnh191a3fmg3e1aub-f4w6r1h1a4m1v8n", pod="openshift-apiserver-7dc759866b-hr75b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2ti4iojordd6hm0nnh191a3fmg3e1aub-f4w6r1h1a4m1v8n/openshift-apiserver-7dc759866b-hr75b has been in a non-ready state for longer than 5 minutes.

alert fired 10 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T21:42:05Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-796b59578b-rbqnk", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-796b59578b-rbqnk has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T21:42:05Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-vr6v5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-vr6v5 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-19T21:42:05Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-9vsw9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-9vsw9 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3pc5270b8h60pst14u078uaplvhlq-q0m7n1d5p4o1u4j", pod="azure-disk-csi-driver-controller-65f5bf78b-s9gq4", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3pc5270b8h60pst14u078uaplvhlq-q0m7n1d5p4o1u4j/azure-disk-csi-driver-controller-65f5bf78b-s9gq4 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-sb8lp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-sb8lp has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-796b59578b-bjj96", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-796b59578b-bjj96 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-796b59578b-w4b2v", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-796b59578b-w4b2v has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-8qtrk", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-8qtrk has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-mkmmq", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-mkmmq has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-19T21:42:06Z
  Ended:       2026-10-19T22:08:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3737472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas", pod="azure-disk-csi-driver-controller-68d8d85f58-rvxrh", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti3p9tdj7jhju8rr4baflukekfp093j-private-kas/azure-disk-csi-driver-controller-68d8d85f58-rvxrh has been in a non-ready state for longer than 5 minutes.

alert fired 13 time(s) (1 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T20:56:48Z
  Ended:       2026-10-19T21:03:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7537536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ti35rfutnepdsfn0vfq6aa39v1svufp-f2i3r5n2t0v2v3v", pod="router-6dbb4f8499-ccrn9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2ti35rfutnepdsfn0vfq6aa39v1svufp-f2i3r5n2t0v2v3v/router-6dbb4f8499-ccrn9 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	13

Affected runs (13)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:24	PR #7547	2112293841544417280
2026-10-19 20:33	PR #7612	2112281126293737472
2026-10-19 20:09	PR #7604	2112274862377537536
2026-10-19 19:03	PR #7602	2112258228648677376
2026-10-19 18:08	PR #7609	2112244635597279232
2026-10-19 17:13	PR #7480	2112230579016568832
2026-10-19 15:37	PR #6920	2112206455762325504
2026-10-19 15:31	PR #7609	2112205019058016256
2026-10-19 15:09	PR #7301	2112199501795037184
2026-10-19 14:18	PR #7556	2112186650028150784
2026-10-19 13:42	PR #7220	2112177636460466176
2026-10-19 12:59	PR #7598	2112166731613999104
2026-10-19 09:43	PR #7582	2112117383177965568

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-20T00:03:40Z
  Ended:       2026-10-20T00:18:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0435200-svc", component="backend", name="ingresscertificate", region="westus3", severity="warning"
  Description: Backend controller workqueue ingresscertificate has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T16:38:15Z
  Ended:       2026-10-19T16:53:16Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5762560-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T15:19:57Z
  Ended:       2026-10-19T15:33:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4950784-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T15:24:05Z
  Ended:       2026-10-19T15:33:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4950784-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-10-19 22:39	PR #7232	2112312824880435200
2026-10-19 15:32	PR #7132	2112205178445762560
2026-10-19 14:11	PR #7480	2112184932984950784
2026-10-19 12:59	PR #7598	2112166731613999104
2026-10-19 08:13	PR #7599	2112094774411399168
2026-10-19 07:05	PR #7597	2112077587801968640

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-d9xldnlstk6j/cluster-candidate-5-0-tlxgzk should provision
Unexpected error:
    <*fmt.wrapError | 0x2c66751baf40>: 
    failed to create HCP cluster cluster-candidate-5-0-tlxgzk: failed waiting for cluster="cluster-candidate-5-0-tlxgzk" in resourcegroup="rg-candidate-5-0-d9xldnlstk6j" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065",
      "name": "6dc91de3-05e9-401b-9139-c466ece83065",
      "status": "Failed",
      "startTime": "2026-10-19T22:04:14.113898084Z",
      "endTime": "2026-10-19T22:33:23.409451279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
            correlation ID: 2a2d3ba7-babe-4fe8-ae4e-5435562f5656
            client request ID: a7fdb086-12e2-40d2-b194-107f4afa279d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-d9xldnlstk6j/cluster-candidate-5-0-tlxgzk should provision
Unexpected error:
    <*fmt.wrapError | 0x2c66751baf40>: 
    failed to create HCP cluster cluster-candidate-5-0-tlxgzk: failed waiting for cluster="cluster-candidate-5-0-tlxgzk" in resourcegroup="rg-candidate-5-0-d9xldnlstk6j" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065",
      "name": "6dc91de3-05e9-401b-9139-c466ece83065",
      "status": "Failed",
      "startTime": "2026-10-19T22:04:14.113898084Z",
      "endTime": "2026-10-19T22:33:23.409451279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dc91de3-05e9-401b-9139-c466ece83065
            correlation ID: 2a2d3ba7-babe-4fe8-ae4e-5435562f5656
            client request ID: a7fdb086-12e2-40d2-b194-107f4afa279d
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xdbc40b14b80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-9ndqj45cmqg2" to finish creating: GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "name": "a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "status": "Failed",
      "startTime": "2026-10-19T18:15:40.641153712Z",
      "endTime": "2026-10-19T18:44:47.329454995Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
            correlation ID: b9b321eb-17a4-4f82-91e4-993a13652950
            client request ID: f220b5f7-1c64-40ab-bc96-b0e5ef27428d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xdbc40b14b80>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-9ndqj45cmqg2" to finish creating: GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "name": "a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9",
      "status": "Failed",
      "startTime": "2026-10-19T18:15:40.641153712Z",
      "endTime": "2026-10-19T18:44:47.329454995Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8442752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1467e3e-7ac2-4094-9be0-5ea4dcdef5e9
            correlation ID: b9b321eb-17a4-4f82-91e4-993a13652950
            client request ID: f220b5f7-1c64-40ab-bc96-b0e5ef27428d
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-bbzfqrxldsr2/cluster-candidate-5-1-vz4k7j should provision
Unexpected error:
    <*fmt.wrapError | 0x399878fce5c0>: 
    failed to create HCP cluster cluster-candidate-5-1-vz4k7j: failed waiting for cluster="cluster-candidate-5-1-vz4k7j" in resourcegroup="rg-candidate-5-1-bbzfqrxldsr2" to finish creating: GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "name": "f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "status": "Failed",
      "startTime": "2026-10-19T16:52:45.887911577Z",
      "endTime": "2026-10-19T17:21:46.629500013Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
            correlation ID: 4cf69c21-e6dd-4c6d-8bd7-3644b7285946
            client request ID: 2f22a1ad-d176-4f31-9493-27c1ef58e861
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-bbzfqrxldsr2/cluster-candidate-5-1-vz4k7j should provision
Unexpected error:
    <*fmt.wrapError | 0x399878fce5c0>: 
    failed to create HCP cluster cluster-candidate-5-1-vz4k7j: failed waiting for cluster="cluster-candidate-5-1-vz4k7j" in resourcegroup="rg-candidate-5-1-bbzfqrxldsr2" to finish creating: GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "name": "f0513279-bd34-4b93-937c-26fa6b8a6aeb",
      "status": "Failed",
      "startTime": "2026-10-19T16:52:45.887911577Z",
      "endTime": "2026-10-19T17:21:46.629500013Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0032768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0513279-bd34-4b93-937c-26fa6b8a6aeb
            correlation ID: 4cf69c21-e6dd-4c6d-8bd7-3644b7285946
            client request ID: 2f22a1ad-d176-4f31-9493-27c1ef58e861
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:03	PR #7602	2112288570218647552
2026-10-19 17:24	PR #7566	2112233538538442752
2026-10-19 15:55	PR #7601	2112210984000032768
2026-10-19 12:28	PR #7603	2112158901787955200
2026-10-19 09:43	PR #7582	2112117383177965568

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:28:50Z
  Ended:       2026-10-19T22:55:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="43f3c967-dd75-47e1-aa51-7f18084ed726"
  Description: Pod ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T22:29:50Z
  Ended:       2026-10-19T22:55:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="43f3c967-dd75-47e1-aa51-7f18084ed726"
  Description: Pod ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:36:54Z
  Ended:       2026-10-19T19:06:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ab10566e-97b4-4b52-af7f-f5418105b4d1"
  Description: Pod ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:36:54Z
  Ended:       2026-10-19T19:06:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ab10566e-97b4-4b52-af7f-f5418105b4d1"
  Description: Pod ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T17:17:20Z
  Ended:       2026-10-19T17:42:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="001c2a8a-fc34-4e11-96e9-d8a9bf0c0d4a"
  Description: Pod ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-19T17:17:20Z
  Ended:       2026-10-19T17:42:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="001c2a8a-fc34-4e11-96e9-d8a9bf0c0d4a"
  Description: Pod ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:03	PR #7602	2112288570218647552
2026-10-19 17:24	PR #7566	2112233538538442752
2026-10-19 15:55	PR #7601	2112210984000032768
2026-10-19 12:28	PR #7603	2112158901787955200
2026-10-19 09:43	PR #7582	2112117383177965568

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:25:56Z
  Ended:       2026-10-19T22:51:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frhvn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T22:25:56Z
  Ended:       2026-10-19T22:51:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frhvn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:34:58Z
  Ended:       2026-10-19T19:02:56Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:35:57Z
  Ended:       2026-10-19T19:06:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T17:14:26Z
  Ended:       2026-10-19T17:38:23Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8l75h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T17:26:26Z
  Ended:       2026-10-19T17:38:23Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0032768-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.136:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032768/providers/microsoft.monitor/accounts/hcps-j0032768", namespace="ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8l75h", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2thvpnjf9f40p719moscorcvlmrkokr9-e5z0m1p8i3n8y2p/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:03	PR #7602	2112288570218647552
2026-10-19 17:24	PR #7566	2112233538538442752
2026-10-19 15:55	PR #7601	2112210984000032768
2026-10-19 15:37	PR #6920	2112206455762325504
2026-10-19 12:28	PR #7603	2112158901787955200

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xbabb28f39e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-zk8hdsmg8dfw" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "name": "7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "status": "Failed",
      "startTime": "2026-10-20T00:39:28.091453996Z",
      "endTime": "2026-10-20T00:58:30.991397546Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
            correlation ID: 8265cd70-ed50-4525-a687-45c965024ae6
            client request ID: b95e9e3d-a0f7-4ee8-8ac7-2ff5d62a1d3a
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xbabb28f39e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-zk8hdsmg8dfw" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "name": "7dd28f77-c5a7-447f-9ff3-55ca2c7472be",
      "status": "Failed",
      "startTime": "2026-10-20T00:39:28.091453996Z",
      "endTime": "2026-10-20T00:58:30.991397546Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7dd28f77-c5a7-447f-9ff3-55ca2c7472be
            correlation ID: 8265cd70-ed50-4525-a687-45c965024ae6
            client request ID: b95e9e3d-a0f7-4ee8-8ac7-2ff5d62a1d3a

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xcdba5b46020>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-9bqdtwnxktt9" to finish creating: GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "name": "f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "status": "Failed",
      "startTime": "2026-10-19T21:20:53.14343965Z",
      "endTime": "2026-10-19T21:40:01.878429551Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
            correlation ID: 85a1b8fd-965d-4893-90c4-5bc86f87328d
            client request ID: 6c4ed168-8339-4ad9-9c7e-dcd04bbc9a76
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xcdba5b46020>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-9bqdtwnxktt9" to finish creating: GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "name": "f0e04584-5bfd-41bb-a2f1-00de7ac45771",
      "status": "Failed",
      "startTime": "2026-10-19T21:20:53.14343965Z",
      "endTime": "2026-10-19T21:40:01.878429551Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1d9g5x1u0o0j2g-np-1-klnmc-mrz7g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1193984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0e04584-5bfd-41bb-a2f1-00de7ac45771
            correlation ID: 85a1b8fd-965d-4893-90c4-5bc86f87328d
            client request ID: 6c4ed168-8339-4ad9-9c7e-dcd04bbc9a76

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x39c1da55a940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-vfc45fcxtxqz" to finish creating: GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "name": "98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "status": "Failed",
      "startTime": "2026-10-19T18:03:22.205099398Z",
      "endTime": "2026-10-19T18:22:25.549095315Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
            correlation ID: 697bbb7b-4488-43cf-b80e-bef1ce56a7c0
            client request ID: 9c6f432c-bb1f-45d1-a1cc-acc4672dbd73
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x39c1da55a940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-vfc45fcxtxqz" to finish creating: GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "name": "98bf5641-ac12-45f2-b24a-9f7eddb62757",
      "status": "Failed",
      "startTime": "2026-10-19T18:03:22.205099398Z",
      "endTime": "2026-10-19T18:22:25.549095315Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-vfc45fcxtxqz--managed/providers/Microsoft.Compute/virtualMachines/z3r1w1e9r2b6c5r-np-1-cpcnn-wsxkm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5859328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/98bf5641-ac12-45f2-b24a-9f7eddb62757
            correlation ID: 697bbb7b-4488-43cf-b80e-bef1ce56a7c0
            client request ID: 9c6f432c-bb1f-45d1-a1cc-acc4672dbd73

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate for 5.1	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-19 23:13	PR #7621	2112321281692012544
2026-10-19 20:10	PR #7463	2112275231841193984
2026-10-19 17:09	PR #6191	2112229674405859328
2026-10-19 03:13	PR #7504	2112019392656052224

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T22:40:56Z
  Ended:       2026-10-19T22:52:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8647552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8647552/providers/microsoft.monitor/accounts/hcps-j8647552", namespace="ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frhvn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ti4bopt6j217c6p864tnvp7ieejcmnb-k4p0h9n4o7o7c1n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T18:49:54Z
  Ended:       2026-10-19T19:02:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T18:52:56Z
  Ended:       2026-10-19T19:03:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8442752-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8442752/providers/microsoft.monitor/accounts/hcps-j8442752", namespace="ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-x7lq7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ti10j9gmvla65fehtd18pglombd8rli-y4w7z9d4n7l1q9v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-19T13:54:50Z
  Ended:       2026-10-19T14:07:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vsdh5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-19T13:55:53Z
  Ended:       2026-10-19T14:07:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7955200-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.25:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7955200/providers/microsoft.monitor/accounts/hcps-j7955200", namespace="ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vsdh5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2thsm2jsmop3m17ptumulcji2sfp5cmr-s4t2f9w2p6b9m4v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:03	PR #7602	2112288570218647552
2026-10-19 17:24	PR #7566	2112233538538442752
2026-10-19 12:28	PR #7603	2112158901787955200
2026-10-19 09:43	PR #7582	2112117383177965568

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x17293bf48060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-qw8hmnl947sz" to finish creating: GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd",
      "name": "2be64b2f-b78e-49f6-addd-04faf33378bd",
      "status": "Failed",
      "startTime": "2026-10-19T23:54:37.149407446Z",
      "endTime": "2026-10-20T00:13:41.899452027Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
            correlation ID: a4d6efb0-533f-4043-ad67-651fe433c379
            client request ID: 6f8a1c34-923e-40e5-a5b3-733c67d8d88f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x17293bf48060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-qw8hmnl947sz" to finish creating: GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd",
      "name": "2be64b2f-b78e-49f6-addd-04faf33378bd",
      "status": "Failed",
      "startTime": "2026-10-19T23:54:37.149407446Z",
      "endTime": "2026-10-20T00:13:41.899452027Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0435200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2be64b2f-b78e-49f6-addd-04faf33378bd
            correlation ID: a4d6efb0-533f-4043-ad67-651fe433c379
            client request ID: 6f8a1c34-923e-40e5-a5b3-733c67d8d88f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2d83e42499e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-85n46zxr29bc" to finish creating: GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da",
      "name": "eb476919-259d-4581-ba0c-41803ac281da",
      "status": "Failed",
      "startTime": "2026-10-19T17:15:51.295694688Z",
      "endTime": "2026-10-19T17:34:52.47411646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
            correlation ID: 2736a12e-0c94-445a-972d-f60f128898b0
            client request ID: 8ae6dbe5-3e36-4ee4-b803-b4d4643b03d4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x2d83e42499e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-85n46zxr29bc" to finish creating: GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da",
      "name": "eb476919-259d-4581-ba0c-41803ac281da",
      "status": "Failed",
      "startTime": "2026-10-19T17:15:51.295694688Z",
      "endTime": "2026-10-19T17:34:52.47411646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2294016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb476919-259d-4581-ba0c-41803ac281da
            correlation ID: 2736a12e-0c94-445a-972d-f60f128898b0
            client request ID: 8ae6dbe5-3e36-4ee4-b803-b4d4643b03d4
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-19 22:39	PR #7232	2112312824880435200
2026-10-19 16:02	PR #7613	2112212933982294016

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x13c01da76240>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x13c01e312060>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x13c01f5045e0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x13c01e3822a0>{
                        msg: "DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x13c01da76240>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x13c01e312060>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x13c01f5045e0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x13c01e3822a0>{
                        msg: "DNS for agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-26v8b.apps.aro.agg-cluster.4cnq.j7435648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0x1ffd80fcc600>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1ffd80c0ac80>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1ffd811f6800>{
                    msg: "DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1ffd80d0e5e0>{
                        msg: "DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0x1ffd80fcc600>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1ffd80c0ac80>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1ffd811f6800>{
                    msg: "DNS for route host agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1ffd80d0e5e0>{
                        msg: "DNS for agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6r5mt.apps.aro.n2r0d3y7o1g9i7q.hc5u.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-19 19:36	PR #7614	2112266775457435648
2026-10-19 15:09	PR #7301	2112199501795037184

failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1

Signal:

provision

none

Full failure examples (2)

time=2026-10-19T19:36:46.694Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-10-19T19:36:47.222Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-19T19:36:47.222Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T19:36:47.440Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-10-19T19:36:47.554Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-10-19T19:36:47.554Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T19:36:47.600Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T19:42:29.171Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T19:42:29.171Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

time=2026-10-19T15:56:31.916Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-10-19T15:56:32.339Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-19T15:56:32.339Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T15:56:32.526Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-10-19T15:56:32.659Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-10-19T15:56:32.659Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T15:56:32.731Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-19T16:02:20.224Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T16:02:20.224Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mitigations (stamp=2)	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-19 19:06	PR #7133	2112259193590255616
2026-10-19 15:24	PR #7133	2112203264580653056

ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: failed to create customer resources for second cluster "basic-hcp-cluster2"
Unexpected error:
    <*fmt.wrapError | 0x3cfd776cc680>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-vrgp2b" in resourcegroup="customer-rg-d7bzdbm7gf6t" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xooij5h335npw' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
            correlation ID: 0edba454-3af2-4d43-98da-9b281d7b5b9e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: failed to create customer resources for second cluster "basic-hcp-cluster2"
Unexpected error:
    <*fmt.wrapError | 0x3cfd776cc680>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-vrgp2b" in resourcegroup="customer-rg-d7bzdbm7gf6t" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-xooij5h335npw' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-d7bzdbm7gf6t/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-vrgp2b/operationStatuses/08584091731103433891
            correlation ID: 0edba454-3af2-4d43-98da-9b281d7b5b9e
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 17:24	PR #7566	2112233538538442752

registry.redhat.io credentials not found in pull-secret file

full failure pattern:

registry.redhat.io credentials not found in pull-secret file

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
    <bool>: false
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	21

Affected runs (21)

Date (UTC)	Associated PR	Prow job
2026-10-19 17:24	PR #7566	2112233538538442752
2026-10-19 17:13	PR #7480	2112230579016568832
2026-10-19 17:09	PR #6191	2112229674405859328
2026-10-19 16:39	PR #7497	2112222204212023296
2026-10-19 16:39	PR #7604	2112222016026185728
2026-10-19 16:38	PR #7556	2112221900917706752
2026-10-19 16:36	PR #7592	2112221482724626432
2026-10-19 16:02	PR #7613	2112212933982294016
2026-10-19 15:59	PR #6866	2112211956654936064
2026-10-19 15:55	PR #7601	2112210984000032768
2026-10-19 15:37	PR #6920	2112206455762325504
2026-10-19 15:32	PR #7132	2112205178445762560
2026-10-19 15:31	PR #7609	2112205019058016256
2026-10-19 15:11	PR #7597	2112199925969195008
2026-10-19 15:09	PR #7301	2112199501795037184
2026-10-19 14:46	PR #7387	2112193616469299200
2026-10-19 14:18	PR #7556	2112186650028150784
2026-10-19 14:11	PR #7480	2112184932984950784
2026-10-19 14:06	PR #6968	2112183526613848064
2026-10-19 13:42	PR #7220	2112177636460466176
2026-10-19 13:26	PR #6718	2112173510980800512

invalid character 'M' looking for beginning of value

full failure pattern:

invalid character 'M' looking for beginning of value

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x3eb77839e738>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x3eb77839e738>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x62872c04378>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x62872c04378>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x25effd432cd8>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:193]: failed to parse pull-secret file
Unexpected error:
    <*json.SyntaxError | 0x25effd432cd8>: 
    invalid character 'M' looking for beginning of value
    {
        msg: "invalid character 'M' looking for beginning of value",
        Offset: 1,
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	9

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-10-19 13:38	PR #7606	2112176673532153856
2026-10-19 13:25	PR #6688	2112173242356600832
2026-10-19 13:05	PR #7604	2112168271166836736
2026-10-19 12:59	PR #7598	2112166731613999104
2026-10-19 12:46	PR #7497	2112163579019923456
2026-10-19 12:28	PR #7603	2112158901787955200
2026-10-19 12:14	PR #7441	2112155408402485248
2026-10-19 12:11	PR #7597	2112154781085601792
2026-10-19 11:29	PR #7065	2112144183836807168

failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1

Signal:

provision

none

Full failure examples (3)

time=2026-10-20T00:14:03.097Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-20T00:14:03.476Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-20T00:14:03.476Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-20T00:14:03.887Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-20T00:14:04.156Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-20T00:14:04.156Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-20T00:14:04.247Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-20T00:15:01.574Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-20T00:15:01.574Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

time=2026-10-19T20:14:27.366Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T20:14:27.872Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T20:14:27.872Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T20:14:28.027Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-19T20:14:28.232Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-19T20:14:28.232Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T20:14:28.292Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-19T20:15:24.446Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T20:15:24.446Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

time=2026-10-19T19:34:54.564Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-19T19:34:54.990Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]"
time=2026-10-19T19:34:54.990Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T19:34:55.299Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-19T19:34:55.456Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-19T19:34:55.456Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T19:34:55.507Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-19T19:35:45.081Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-19T19:35:45.081Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"

Contributing tests (2)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=1)	5
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=2)	4

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-10-19 23:18	PR #7451	2112322496756715520
2026-10-19 19:52	PR #7451	2112270689380601856
2026-10-19 19:10	PR #7451	2112260172159127552
2026-10-19 17:52	PR #7451	2112240367163150336
2026-10-19 17:05	PR #7451	2112228565553516544
2026-10-19 16:05	PR #7608	2112213551379648512
2026-10-19 15:40	PR #7451	2112207281033580544
2026-10-19 14:54	PR #7451	2112195721783414784
2026-10-19 06:56	PR #7451	2112075402955460608

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1d0051281e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6h69n8f46csz" to finish creating: GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f",
      "name": "65c67213-a430-416b-a8e1-9c576702841f",
      "status": "Failed",
      "startTime": "2026-10-19T23:03:54.135232982Z",
      "endTime": "2026-10-19T23:22:56.513764235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
            correlation ID: d01ca859-c416-4ccc-bd23-51544b0af025
            client request ID: 937df210-9f80-46da-9e07-0166c55c864d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1d0051281e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6h69n8f46csz" to finish creating: GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f",
      "name": "65c67213-a430-416b-a8e1-9c576702841f",
      "status": "Failed",
      "startTime": "2026-10-19T23:03:54.135232982Z",
      "endTime": "2026-10-19T23:22:56.513764235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6h69n8f46csz--managed/providers/Microsoft.Compute/virtualMachines/q9d8h1b2n4i4y9z-np-1-wb6tr-5xthm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5885568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/65c67213-a430-416b-a8e1-9c576702841f
            correlation ID: d01ca859-c416-4ccc-bd23-51544b0af025
            client request ID: 937df210-9f80-46da-9e07-0166c55c864d
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-24dvjs" for cluster "cluster-hs-pre-24dvjs"
Unexpected error:
    <*fmt.wrapError | 0x3d397c5fe960>: 
    failed to create NodePool np-24dvjs: failed waiting for nodepool="np-24dvjs" for cluster "cluster-hs-pre-24dvjs" in resourcegroup="rg-hs-pre--qmdx974n6w26" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "name": "8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "status": "Failed",
      "startTime": "2026-10-19T22:14:24.493403189Z",
      "endTime": "2026-10-19T22:33:34.182492247Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
            correlation ID: 33ce383b-ea0f-4d30-8902-6098f7bbfeb8
            client request ID: 9ce1f451-6c72-4c5f-8183-8c4490de781d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-24dvjs" for cluster "cluster-hs-pre-24dvjs"
Unexpected error:
    <*fmt.wrapError | 0x3d397c5fe960>: 
    failed to create NodePool np-24dvjs: failed waiting for nodepool="np-24dvjs" for cluster "cluster-hs-pre-24dvjs" in resourcegroup="rg-hs-pre--qmdx974n6w26" to finish creating: GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "name": "8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0",
      "status": "Failed",
      "startTime": "2026-10-19T22:14:24.493403189Z",
      "endTime": "2026-10-19T22:33:34.182492247Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--qmdx974n6w26-24dvjs--managed/providers/Microsoft.Compute/virtualMachines/w6l5b9y4j0d7a0b-np-24dvjs-sb457-th5dq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8647552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8c4bd3c2-dba4-4fd4-adb1-10b5c21cbdf0
            correlation ID: 33ce383b-ea0f-4d30-8902-6098f7bbfeb8
            client request ID: 9ce1f451-6c72-4c5f-8183-8c4490de781d
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x3b24ac0020c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-p69pvshqvq9p" to finish creating: GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "name": "c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "status": "Failed",
      "startTime": "2026-10-19T15:56:13.653725725Z",
      "endTime": "2026-10-19T16:15:18.87162238Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
            correlation ID: 5cc9d41d-adc2-4131-802d-17d3ab484012
            client request ID: b9e380f6-f1bc-4da7-bbfb-dbb4f50cb471
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x3b24ac0020c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-p69pvshqvq9p" to finish creating: GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "name": "c4f1b382-5a50-42f4-93c5-591e1ed0f369",
      "status": "Failed",
      "startTime": "2026-10-19T15:56:13.653725725Z",
      "endTime": "2026-10-19T16:15:18.87162238Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-p69pvshqvq9p--managed/providers/Microsoft.Compute/virtualMachines/r0m5h3v1x6r8h3a-np-1-4x7nf-kblcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8627328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f1b382-5a50-42f4-93c5-591e1ed0f369
            correlation ID: 5cc9d41d-adc2-4131-802d-17d3ab484012
            client request ID: b9e380f6-f1bc-4da7-bbfb-dbb4f50cb471
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-19 21:57	PR #7480	2112302045275885568
2026-10-19 21:03	PR #7602	2112288570218647552
2026-10-19 14:35	PR #7250	2112190987718627328

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

full failure pattern:

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc80b6d518e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc80b6d518e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xaeb08df4660>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xaeb08df4660>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.009s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x207ea69dfda0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:491]: Timed out after 600.009s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x207ea69dfda0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-19 23:13	PR #7621	2112321281692012544
2026-10-19 23:12	PR #7463	2112321203636015104
2026-10-19 22:39	PR #7232	2112312824880435200

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-sjxzmq"
Unexpected error:
    <*fmt.wrapError | 0x3e0c69717cc0>: 
    DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3e0c6a7f4740>{
            msg: "DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-sjxzmq"
Unexpected error:
    <*fmt.wrapError | 0x3e0c69717cc0>: 
    DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3e0c6a7f4740>{
            msg: "DNS for agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mbxj4.apps.aro.i3o6o2t9x9d5w5r.jkfh.j5037184.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1a6b532d3880>: 
    DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1a6b532d22c0>{
            msg: "DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1a6b532d3880>: 
    DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1a6b532d22c0>{
            msg: "DNS for agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dhkfh.apps.aro.v20261001.aza8.j6836736.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-2qp7lk"
Unexpected error:
    <*fmt.wrapError | 0xd92a3e87160>: 
    DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xd92a3f627c0>{
            msg: "DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-2qp7lk"
Unexpected error:
    <*fmt.wrapError | 0xd92a3e87160>: 
    DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xd92a3f627c0>{
            msg: "DNS for agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5vnqx.apps.aro.x6a5q8q6c4f1a0y.23a8.j7201408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-19 15:09	PR #7301	2112199501795037184
2026-10-19 13:05	PR #7604	2112168271166836736
2026-10-19 11:10	PR #7601	2112139306637201408

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:437]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x340bb99d4c0>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rhzhzsd2dvs2" to finish creating: GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de",
      "name": "1520252c-b285-49f1-8706-c82f9e3a45de",
      "status": "Failed",
      "startTime": "2026-10-19T20:28:35.166007052Z",
      "endTime": "2026-10-19T20:47:38.665697527Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
            correlation ID: da09f0ee-1b1c-4dd0-986e-72ed6522e248
            client request ID: 773c74d7-3374-4a99-9619-c2776f1df3e0
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:437]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x340bb99d4c0>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rhzhzsd2dvs2" to finish creating: GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de",
      "name": "1520252c-b285-49f1-8706-c82f9e3a45de",
      "status": "Failed",
      "startTime": "2026-10-19T20:28:35.166007052Z",
      "endTime": "2026-10-19T20:47:38.665697527Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine s5p6y5u2k0u9n5z-worker-dlpqz-bfjsj: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7435648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1520252c-b285-49f1-8706-c82f9e3a45de
            correlation ID: da09f0ee-1b1c-4dd0-986e-72ed6522e248
            client request ID: 773c74d7-3374-4a99-9619-c2776f1df3e0
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1051bd7af500>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-p7tdc2" in resourcegroup="rg-np-version-upgrade-p7tdc2-lnnmsq2r4w9h" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "name": "70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "status": "Failed",
      "startTime": "2026-10-19T17:02:00.685391332Z",
      "endTime": "2026-10-19T17:21:08.269829146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
            correlation ID: fb9e2f48-428d-4f30-8c61-d916aa767023
            client request ID: 93ac57ce-9b86-4ca4-8a14-339c496ee50f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x1051bd7af500>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-p7tdc2" in resourcegroup="rg-np-version-upgrade-p7tdc2-lnnmsq2r4w9h" to finish creating: GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "name": "70410933-3edd-4cf6-ad9e-b8f9f2dc028a",
      "status": "Failed",
      "startTime": "2026-10-19T17:02:00.685391332Z",
      "endTime": "2026-10-19T17:21:08.269829146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3s0j2g6n1a2d0s-npupgrade-4-20-xl6sw-4whr7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2325504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70410933-3edd-4cf6-ad9e-b8f9f2dc028a
            correlation ID: fb9e2f48-428d-4f30-8c61-d916aa767023
            client request ID: 93ac57ce-9b86-4ca4-8a14-339c496ee50f
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-19 19:36	PR #7614	2112266775457435648
2026-10-19 15:37	PR #6920	2112206455762325504

ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the r...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the request.; provider Microsoft.Network

Signal:

provision

none

Full failure examples (2)

time=2026-10-19T18:44:25.577Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-19T18:44:27.375Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-19T18:44:29.162Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=6167d0252600ec817c994890bab60d9173f655840adcabf122067da893127d1f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8937216-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6167d0252600ec817c994890bab60d9173f655840adcabf122067da893127d1f
time=2026-10-19T18:53:01.241Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Resources/deployments/6167d0252600ec817c994890bab60d9173f655840adcabf122067da893127d1f/operationStatuses/08584091714172679540\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Resources/deployments/profile-w5ahaumc2w6ei\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j8937216-svc/resourceAssociations/nsp-j8937216-svc-ra-d5a635c9-43bf-5de7-aaf2-5232b9ebe5d0\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8937216-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j8937216-svc/resourceAssociations/nsp-j8937216-svc-ra-d5a635c9-43bf-5de7-aaf2-5232b9ebe5d0, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-19T14:43:12.749Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-19T14:43:13.235Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1
time=2026-10-19T14:43:13.684Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 deployment=299b7403a208c1c427ae8ebbcd18bb4f4fb4ff513a91ec66ec3b8e76bf4affb0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1534464-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F299b7403a208c1c427ae8ebbcd18bb4f4fb4ff513a91ec66ec3b8e76bf4affb0
time=2026-10-19T14:43:43.960Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Resources/deployments/299b7403a208c1c427ae8ebbcd18bb4f4fb4ff513a91ec66ec3b8e76bf4affb0/operationStatuses/08584091858920261215\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Resources/deployments/profile-xcn2acurowiq2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j1534464-mgmt-1/resourceAssociations/nsp-j1534464-mgmt-1-ra-66bb4858-d2d4-5bd8-8103-db26bfc7bd94\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1534464-mgmt-1/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j1534464-mgmt-1/resourceAssociations/nsp-j1534464-mgmt-1-ra-66bb4858-d2d4-5bd8-8103-db26bfc7bd94, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (2)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=1)	1
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-19 18:22	PR #7602	2112247946668937216
2026-10-19 13:58	PR #7592	2112181488601534464

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-bkqnpw2l8zf5/cluster-candidate-5-0-ccpvpc should provision
Unexpected error:
    <*fmt.wrapError | 0x3d23d4310dc0>: 
    failed to create HCP cluster cluster-candidate-5-0-ccpvpc: failed waiting for cluster="cluster-candidate-5-0-ccpvpc" in resourcegroup="rg-candidate-5-0-bkqnpw2l8zf5" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c",
      "name": "44c0dc0d-efb0-45b9-885c-ff622849913c",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:41.722981857Z",
      "endTime": "2026-10-19T11:00:44.465792588Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
            correlation ID: 77467a09-84cb-420c-a3bc-027f97f67194
            client request ID: 2b188110-77bd-4bc4-9d94-27a812b7e09a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-bkqnpw2l8zf5/cluster-candidate-5-0-ccpvpc should provision
Unexpected error:
    <*fmt.wrapError | 0x3d23d4310dc0>: 
    failed to create HCP cluster cluster-candidate-5-0-ccpvpc: failed waiting for cluster="cluster-candidate-5-0-ccpvpc" in resourcegroup="rg-candidate-5-0-bkqnpw2l8zf5" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c",
      "name": "44c0dc0d-efb0-45b9-885c-ff622849913c",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:41.722981857Z",
      "endTime": "2026-10-19T11:00:44.465792588Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/44c0dc0d-efb0-45b9-885c-ff622849913c
            correlation ID: 77467a09-84cb-420c-a3bc-027f97f67194
            client request ID: 2b188110-77bd-4bc4-9d94-27a812b7e09a
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xd36763f9420>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-q6wh6zbcv2x2" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997",
      "name": "42423002-b716-49bf-b44c-2e1fd4733997",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.508047813Z",
      "endTime": "2026-10-19T11:00:34.28001697Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
            correlation ID: 1aed53cb-56a1-406f-8bbf-9de9f16e4e1c
            client request ID: 90f415c0-66a9-4a24-b9de-2db2faf18c9c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xd36763f9420>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-q6wh6zbcv2x2" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997",
      "name": "42423002-b716-49bf-b44c-2e1fd4733997",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.508047813Z",
      "endTime": "2026-10-19T11:00:34.28001697Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42423002-b716-49bf-b44c-2e1fd4733997
            correlation ID: 1aed53cb-56a1-406f-8bbf-9de9f16e4e1c
            client request ID: 90f415c0-66a9-4a24-b9de-2db2faf18c9c
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 09:43	PR #7582	2112117383177965568

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:184]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0x1e0c6d55a40>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1e0c6d55a20>{
            msg: "failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1e0c6d55a00>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:184]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0x1e0c6d55a40>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1e0c6d55a20>{
            msg: "failed to get DaemonSet kube-system/global-pull-secret-syncer: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1e0c6d55a00>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 07:05	PR #7597	2112077587801968640

| take 101, Post "<url>": context deadline exceeded

full failure pattern:

| take 101, Post "<url>": context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cde6af4c2e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-19T23:40:50.5353725Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1cde6af4c2c0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1cde6a0b6140>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1cde6af4c2a0>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1cde6c048750>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cde6af4c2e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-19T23:40:50.5353725Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1cde6af4c2c0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1cde6a0b6140>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1cde6af4c2a0>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hx6p2lg8h8cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-19T23:40:50.5353725Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1cde6c048750>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 22:39	PR #7232	2112312824880435200

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x3d4749b3c078>: 
    resource group "customer-rg-j9j4x64vc846": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-j9j4x64vc846" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867",
      "name": "060bb367-1bd8-42f9-b696-549f7d9fb867",
      "status": "Failed",
      "startTime": "2026-10-19T20:02:54.979353355Z",
      "endTime": "2026-10-19T20:27:00.270376787Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ti2ch5ekujd757n62ntgfdpp1p6fpca still exists (deletion dispatched at 2026-10-19T20:02:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
            correlation ID: 1b181c6c-5659-46ac-a08f-39fecbba1c31
            client request ID: 190e259b-25d3-414f-bd78-f91c53d9c681
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x3d4749b3c078>: 
    resource group "customer-rg-j9j4x64vc846": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-j9j4x64vc846" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867",
      "name": "060bb367-1bd8-42f9-b696-549f7d9fb867",
      "status": "Failed",
      "startTime": "2026-10-19T20:02:54.979353355Z",
      "endTime": "2026-10-19T20:27:00.270376787Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ti2ch5ekujd757n62ntgfdpp1p6fpca still exists (deletion dispatched at 2026-10-19T20:02:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/060bb367-1bd8-42f9-b696-549f7d9fb867
            correlation ID: 1b181c6c-5659-46ac-a08f-39fecbba1c31
            client request ID: 190e259b-25d3-414f-bd78-f91c53d9c681
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 19:10	PR #7497	2112260148863963136

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x424386240f0>: 
    resource group "delayed-rbac-9xs9kgbdmnwv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-9xs9kgbdmnwv" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "name": "2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "status": "Failed",
      "startTime": "2026-10-19T19:53:00.471686176Z",
      "endTime": "2026-10-19T20:17:09.335861904Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
            correlation ID: e5bf541d-883b-4a45-98e4-199dc9b858a9
            client request ID: 39529ecb-f70a-4c21-a2de-abd0e5ac1e6b
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x424386240f0>: 
    resource group "delayed-rbac-9xs9kgbdmnwv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-9xs9kgbdmnwv" to finish deleting: GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "name": "2dfe8bfe-9c49-4dad-9e91-a524833d6d27",
      "status": "Failed",
      "startTime": "2026-10-19T19:53:00.471686176Z",
      "endTime": "2026-10-19T20:17:09.335861904Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3963136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2dfe8bfe-9c49-4dad-9e91-a524833d6d27
            correlation ID: e5bf541d-883b-4a45-98e4-199dc9b858a9
            client request ID: 39529ecb-f70a-4c21-a2de-abd0e5ac1e6b
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 19:10	PR #7497	2112260148863963136

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xbe67dc5ca00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-gs2cw8rqvtsp" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "name": "9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "status": "Failed",
      "startTime": "2026-10-19T10:33:35.601658411Z",
      "endTime": "2026-10-19T11:02:44.344881038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, csi-snapshot-controller-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, ingress-operator, olm-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, kube-controller-manager, openshift-controller-manager, cluster-network-operator, redhat-marketplace-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-policy-controller, azure-cloud-controller-manager, packageserver, cluster-version-operator, openshift-apiserver, cluster-autoscaler, machine-approver, capi-provider, catalog-operator, community-operators-catalog, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 2 unavailable replicas, kube-controller-manager deployment has 2 unavailable replicas, kube-scheduler deployment has 2 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
            correlation ID: ebcf2a7d-e9a4-4dbb-b982-a31f966e3404
            client request ID: d11643b0-b580-4f4f-a16e-c60b01f474c0
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xbe67dc5ca00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-gs2cw8rqvtsp" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "name": "9c4a9080-1915-4d6c-8c88-a011da6e522c",
      "status": "Failed",
      "startTime": "2026-10-19T10:33:35.601658411Z",
      "endTime": "2026-10-19T11:02:44.344881038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, csi-snapshot-controller-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, ingress-operator, olm-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator, kube-controller-manager, openshift-controller-manager, cluster-network-operator, redhat-marketplace-catalog, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-policy-controller, azure-cloud-controller-manager, packageserver, cluster-version-operator, openshift-apiserver, cluster-autoscaler, machine-approver, capi-provider, catalog-operator, community-operators-catalog, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 2 unavailable replicas, kube-controller-manager deployment has 2 unavailable replicas, kube-scheduler deployment has 2 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 3 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c4a9080-1915-4d6c-8c88-a011da6e522c
            correlation ID: ebcf2a7d-e9a4-4dbb-b982-a31f966e3404
            client request ID: d11643b0-b580-4f4f-a16e-c60b01f474c0
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 09:43	PR #7582	2112117383177965568

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional token... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xacc0f2d2440>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-jlnfb8x8vq5b" to finish creating: GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8",
      "name": "d0f38151-9c16-4736-8305-cfdc47700fc8",
      "status": "Failed",
      "startTime": "2026-10-19T13:29:55.150212587Z",
      "endTime": "2026-10-19T13:48:55.883771022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-fgcc4 to exist\nMachine ephemeral-disk-ephemeral-np-pscp5-w4psz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-w4psz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: Failed: interfaces failed to create or update. err: failed to get existing resource ephemeral-osdisk-jlnfb8x8vq5b--managed/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Network/networkInterfaces/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '10' seconds after additional token... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
            correlation ID: 346dd7ce-e606-492f-925a-aa985115ed46
            client request ID: 9eb4247b-08f9-4cb3-a183-7df951ef2271
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xacc0f2d2440>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-jlnfb8x8vq5b" to finish creating: GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8",
      "name": "d0f38151-9c16-4736-8305-cfdc47700fc8",
      "status": "Failed",
      "startTime": "2026-10-19T13:29:55.150212587Z",
      "endTime": "2026-10-19T13:48:55.883771022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-fgcc4 to exist\nMachine ephemeral-disk-ephemeral-np-pscp5-w4psz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-pscp5-w4psz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-pscp5-fgcc4: Failed: interfaces failed to create or update. err: failed to get existing resource ephemeral-osdisk-jlnfb8x8vq5b--managed/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-jlnfb8x8vq5b--managed/providers/Microsoft.Network/networkInterfaces/ephemeral-disk-ephemeral-np-pscp5-fgcc4-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '10' seconds after additional token... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7955200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d0f38151-9c16-4736-8305-cfdc47700fc8
            correlation ID: 346dd7ce-e606-492f-925a-aa985115ed46
            client request ID: 9eb4247b-08f9-4cb3-a183-7df951ef2271
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 12:28	PR #7603	2112158901787955200

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1fe322c7e500>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2qqsdjg4hsrq" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "name": "7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "status": "Failed",
      "startTime": "2026-10-20T00:36:56.91702797Z",
      "endTime": "2026-10-20T00:55:57.103083611Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh to exist\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-gl748: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-gl748 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-2qqsdjg4hsrq--managed/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Network/networkInterfaces/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to htt... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
            correlation ID: 2083ceae-e353-4ba6-a1e3-ff92ee472926
            client request ID: 9c9e7813-8848-4608-b832-b1dbe65ca4ad
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1fe322c7e500>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2qqsdjg4hsrq" to finish creating: GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "name": "7b17b2a2-ba01-48a2-bbf6-109c6c980ff7",
      "status": "Failed",
      "startTime": "2026-10-20T00:36:56.91702797Z",
      "endTime": "2026-10-20T00:55:57.103083611Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh to exist\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-gl748: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Compute/virtualMachines/d8b7p0j2z4w5h6j-np-1-j5lml-gl748 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d8b7p0j2z4w5h6j-np-1-j5lml-6lglh: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-2qqsdjg4hsrq--managed/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-2qqsdjg4hsrq--managed/providers/Microsoft.Network/networkInterfaces/d8b7p0j2z4w5h6j-np-1-j5lml-6lglh-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5157a63c-87dc-4680-8f46-954b46399bdc' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to htt... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2012544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7b17b2a2-ba01-48a2-bbf6-109c6c980ff7
            correlation ID: 2083ceae-e353-4ba6-a1e3-ff92ee472926
            client request ID: 9c9e7813-8848-4608-b832-b1dbe65ca4ad
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 23:13	PR #7621	2112321281692012544

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeDoesNotExist: Node does not exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Deleting: virtualmachine deleting; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3f88271a0ea0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2lvnhrzd5g7t" to finish creating: GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "name": "9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "status": "Failed",
      "startTime": "2026-10-19T16:38:57.011707926Z",
      "endTime": "2026-10-19T16:58:00.847122787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: NodeDoesNotExist: Node does not exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: Deleting: virtualmachine deleting\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
            correlation ID: 78955e10-7740-4b87-a377-073a6ecddbd0
            client request ID: a8ccc39c-63f9-475d-8c01-623fcd9a0f40
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3f88271a0ea0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2lvnhrzd5g7t" to finish creating: GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "name": "9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9",
      "status": "Failed",
      "startTime": "2026-10-19T16:38:57.011707926Z",
      "endTime": "2026-10-19T16:58:00.847122787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: NodeDoesNotExist: Node does not exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p7u4k9n2d7l9v9q-autoscale-noaz-v7npk-lb6bv: Deleting: virtualmachine deleting\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5762560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9b5197c8-64c6-49b3-b12c-d01b0a5ee1e9
            correlation ID: 78955e10-7740-4b87-a377-073a6ecddbd0
            client request ID: a8ccc39c-63f9-475d-8c01-623fcd9a0f40
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 15:32	PR #7132	2112205178445762560

ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDB

Signal:

provision

none

Full failure examples (1)

time=2026-10-19T23:00:45.350Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-19T23:00:46.914Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-19T23:00:47.803Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f66b53e6663a909706c4cc56f884e08afa3f703badb52eda8425598d78f027f5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1281792%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff66b53e6663a909706c4cc56f884e08afa3f703badb52eda8425598d78f027f5
time=2026-10-19T23:08:19.085Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1281792/providers/Microsoft.Resources/deployments/f66b53e6663a909706c4cc56f884e08afa3f703badb52eda8425598d78f027f5/operationStatuses/08584091560383429326\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1281792/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1281792/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j1281792\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n                \\\"message\\\": \\\"Database account creation failed. Operation Id: 84ce1c4b-104f-4d59-ba97-ca7f687e5f6a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 50b97f78-d517-42ba-b6b3-2e3807c11c56, Request URI: /apps/56b56b41-1aa1-47de-9428-4a9b9ccbfcc7/services/5492004a-a82d-4653-81b2-013acecf0ba5/partitions/4d300830-d333-45bc-8ad6-8d4a8244255d/replicas/134308569966594579s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-10-19T23:06:28.2874662Z, RequestEndTime: 2026-10-19T23:06:33.6942116Z,  Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:05:35.8111686Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.582,\\\\\\\"memory\\\\\\\":21738860.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0143,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:05:45.8134136Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.599,\\\\\\\"memory\\\\\\\":21747896.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0589,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":27},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:05:55.8162151Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.543,\\\\\\\"memory\\\\\\\":21739028.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0371,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:06:05.8318538Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.956,\\\\\\\"memory\\\\\\\":21715984.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.3135,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:06:15.8338344Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.371,\\\\\\\"memory\\\\\\\":21721028.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0646,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-19T23:06:25.8342074Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.996,\\\\\\\"memory\\\\\\\":21715496.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0674,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32}]}\\\\r\\\\nRequestStart: 2026-10-19T23:06:28.2874662Z; ResponseTime: 2026-10-19T23:06:33.6942116Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-centralus1-be312.documents.azure.com:11303/apps/56b56b41-1aa1-47de-9428-4a9b9ccbfcc7/services/5492004a-a82d-4653-81b2-013acecf0ba5/partitions/4d300830-d333-45bc-8ad6-8d4a8244255d/replicas/134308569966594579s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5410.799, ActivityId: 50b97f78-d517-42ba-b6b3-2e3807c11c56, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11303 | status: Connected | lkt: 10/19/2026 11:06:33 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2874662Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0013},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2874675Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0046},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2874721Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0625},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:28.2875346Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5411.5582},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:33.6990928Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0545},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-19T23:06:33.6991473Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-10-19T23:04:15.0012932Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-10-19T23:04:15.0012932Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-10-19T23:04:16.0332114Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":867,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 22:33	PR #7621	2112311076581281792

failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded

full failure pattern:

failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-19T15:30:44.609Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-19T15:30:46.080Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-19T15:30:46.080Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-19T15:30:46.888Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-19T15:30:46.889Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8401536-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-19T15:30:46.889Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j8401536-svc
time=2026-10-19T15:45:50.026Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 14:59	PR #7506	2112197072718401536

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-ld4f2qxz9tlh/cluster-candidate-4-23-prg9qx should provision
Unexpected error:
    <*fmt.wrapError | 0x103e49a6a6a0>: 
    failed to create HCP cluster cluster-candidate-4-23-prg9qx: failed waiting for cluster="cluster-candidate-4-23-prg9qx" in resourcegroup="rg-candidate-4-23-ld4f2qxz9tlh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900",
      "name": "87105fe8-f597-4050-9513-e5fec5cf8900",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.32598821Z",
      "endTime": "2026-10-19T11:00:34.343098485Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, cluster-autoscaler, cluster-image-registry-operator, redhat-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, catalog-operator, kube-storage-version-migrator, hosted-cluster-config-operator, olm-collect-profiles, konnectivity-agent, kube-scheduler, machine-approver, cluster-storage-operator, certified-operators-catalog, cluster-node-tuning-operator, community-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, kube-controller-manager, olm-operator, control-plane-pki-operator, ingress-operator, cluster-version-operator, openshift-apiserver, cluster-policy-controller, dns-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
            correlation ID: fc208c85-0c22-4fd2-b0d8-868115a21156
            client request ID: 2c46da4c-f387-4bd8-8b83-56d49c4e1b76
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-ld4f2qxz9tlh/cluster-candidate-4-23-prg9qx should provision
Unexpected error:
    <*fmt.wrapError | 0x103e49a6a6a0>: 
    failed to create HCP cluster cluster-candidate-4-23-prg9qx: failed waiting for cluster="cluster-candidate-4-23-prg9qx" in resourcegroup="rg-candidate-4-23-ld4f2qxz9tlh" to finish creating: GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900",
      "name": "87105fe8-f597-4050-9513-e5fec5cf8900",
      "status": "Failed",
      "startTime": "2026-10-19T10:31:31.32598821Z",
      "endTime": "2026-10-19T11:00:34.343098485Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, cluster-autoscaler, cluster-image-registry-operator, redhat-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, catalog-operator, kube-storage-version-migrator, hosted-cluster-config-operator, olm-collect-profiles, konnectivity-agent, kube-scheduler, machine-approver, cluster-storage-operator, certified-operators-catalog, cluster-node-tuning-operator, community-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, kube-controller-manager, olm-operator, control-plane-pki-operator, ingress-operator, cluster-version-operator, openshift-apiserver, cluster-policy-controller, dns-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7965568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/87105fe8-f597-4050-9513-e5fec5cf8900
            correlation ID: fc208c85-0c22-4fd2-b0d8-868115a21156
            client request ID: 2c46da4c-f387-4bd8-8b83-56d49c4e1b76
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 09:43	PR #7582	2112117383177965568

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x106d601dad60>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-dgl2pkrhlxx4" to finish creating: GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "name": "87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "status": "Failed",
      "startTime": "2026-10-19T14:24:33.557951156Z",
      "endTime": "2026-10-19T14:43:40.561544029Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine des-encrypt-des-np-rhpbh-rrqrt: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine des-encrypt-des-np-rhpbh-rrqrt: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
            correlation ID: 504c59a4-77a5-4f21-a6c2-acde02224bf6
            client request ID: fb1b6d77-0e4c-4919-b3ea-36fd2e07f63c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x106d601dad60>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-dgl2pkrhlxx4" to finish creating: GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "name": "87cddd9b-5ff6-4112-815a-afc1ac87bb65",
      "status": "Failed",
      "startTime": "2026-10-19T14:24:33.557951156Z",
      "endTime": "2026-10-19T14:43:40.561544029Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine des-encrypt-des-np-rhpbh-rrqrt: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine des-encrypt-des-np-rhpbh-rrqrt: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3999104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/87cddd9b-5ff6-4112-815a-afc1ac87bb65
            correlation ID: 504c59a4-77a5-4f21-a6c2-acde02224bf6
            client request ID: fb1b6d77-0e4c-4919-b3ea-36fd2e07f63c
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-19 12:59	PR #7598	2112166731613999104