fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x152c481045e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s55djgjncgbr" to finish creating: GET https://rp.j5664384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x152c481045e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s55djgjncgbr" to finish creating: GET https://rp.j5664384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0xc0013d1ed0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/services-j7538560"
        }
    }
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0xc0013d1ed0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/services-j7538560"
        }
    }
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz"
Unexpected error:
    <*fmt.wrapError | 0xc001182600>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz" in resourcegroup="rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt" to finish creating: GET https://rp.j6364800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "name": "6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "status": "Failed",
      "startTime": "2026-10-03T00:01:24.493949107Z",
      "endTime": "2026-10-03T00:20:29.737570785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz"
Unexpected error:
    <*fmt.wrapError | 0xc001182600>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz" in resourcegroup="rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt" to finish creating: GET https://rp.j6364800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "name": "6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "status": "Failed",
      "startTime": "2026-10-03T00:01:24.493949107Z",
      "endTime": "2026-10-03T00:20:29.737570785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:06:25Z
  Ended:       2026-10-03T00:13:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6364800-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:07:24Z
  Ended:       2026-10-03T00:19:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7564928-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T00:08:27Z
  Ended:       2026-10-03T00:27:27Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7564928-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 9 time(s) (1 unknown, 8 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:22:54Z
  Ended:       2026-10-03T00:28:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7564928-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6vba5aukt21s79ka316m1hqn628hho-s4a5z1v1k7m8i6h", pod="openshift-apiserver-64ddcdd5c4-tjv4j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6vba5aukt21s79ka316m1hqn628hho-s4a5z1v1k7m8i6h/openshift-apiserver-64ddcdd5c4-tjv4j has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:19:29Z
  Ended:       2026-10-03T00:25:28Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9271680-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9271680-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-03T00:35:30Z
  Ended:       2026-10-03T00:41:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9271680-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9271680-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5fqc8n" on minor version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000b34440>: 
    failed waiting for hcpCluster="cluster-zstream-4-22-5fqc8n" in resourcegroup="rg-zstream-upgrade-4-22-tqtxbbsf75hl" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "name": "30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "status": "Failed",
      "startTime": "2026-10-02T23:01:14.974688957Z",
      "endTime": "2026-10-02T23:45:16.632504331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5fqc8n" on minor version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000b34440>: 
    failed waiting for hcpCluster="cluster-zstream-4-22-5fqc8n" in resourcegroup="rg-zstream-upgrade-4-22-tqtxbbsf75hl" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "name": "30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "status": "Failed",
      "startTime": "2026-10-02T23:01:14.974688957Z",
      "endTime": "2026-10-02T23:45:16.632504331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fskm94" on minor version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a32b40>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-fskm94" in resourcegroup="rg-zstream-upgrade-4-20-7d7sxcc5nbp4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "name": "0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:38.87341489Z",
      "endTime": "2026-10-02T23:43:46.612931387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fskm94" on minor version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a32b40>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-fskm94" in resourcegroup="rg-zstream-upgrade-4-20-7d7sxcc5nbp4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "name": "0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:38.87341489Z",
      "endTime": "2026-10-02T23:43:46.612931387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-xxdkbk" on minor version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc00060c400>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-xxdkbk" in resourcegroup="rg-zstream-upgrade-4-21-fkvp7fgld7fx" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866",
      "name": "3e466c8d-b439-4e8d-9c16-270890d65866",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:19.732496702Z",
      "endTime": "2026-10-02T23:43:26.633598885Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-xxdkbk" on minor version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc00060c400>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-xxdkbk" in resourcegroup="rg-zstream-upgrade-4-21-fkvp7fgld7fx" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866",
      "name": "3e466c8d-b439-4e8d-9c16-270890d65866",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:19.732496702Z",
      "endTime": "2026-10-02T23:43:26.633598885Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*fmt.wrapError | 0xc000a6f800>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-xzgkv9b984lz" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "name": "bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:51.980435313Z",
      "endTime": "2026-10-02T23:08:01.293538008Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*fmt.wrapError | 0xc000a6f800>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-xzgkv9b984lz" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "name": "bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:51.980435313Z",
      "endTime": "2026-10-02T23:08:01.293538008Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-qrrxbq" on minor version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000b7e340>: 
    failed waiting for hcpCluster="cluster-zstream-5-0-qrrxbq" in resourcegroup="rg-zstream-upgrade-5-0-bnn4f57kv77n" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626",
      "name": "c7191531-446c-4865-aac9-dd2569450626",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:13.4607593Z",
      "endTime": "2026-10-02T23:42:16.686387844Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-qrrxbq" on minor version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000b7e340>: 
    failed waiting for hcpCluster="cluster-zstream-5-0-qrrxbq" in resourcegroup="rg-zstream-upgrade-5-0-bnn4f57kv77n" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626",
      "name": "c7191531-446c-4865-aac9-dd2569450626",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:13.4607593Z",
      "endTime": "2026-10-02T23:42:16.686387844Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-59v6sn" on minor version 4.23
Unexpected error:
    <*fmt.wrapError | 0xc000be1060>: 
    failed waiting for hcpCluster="cluster-zstream-4-23-59v6sn" in resourcegroup="rg-zstream-upgrade-4-23-dwdrgs9rpkt4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32",
      "name": "f2c90638-d31c-4819-aa00-1f73910cfe32",
      "status": "Failed",
      "startTime": "2026-10-02T23:00:29.412121269Z",
      "endTime": "2026-10-02T23:44:36.589759019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-59v6sn" on minor version 4.23
Unexpected error:
    <*fmt.wrapError | 0xc000be1060>: 
    failed waiting for hcpCluster="cluster-zstream-4-23-59v6sn" in resourcegroup="rg-zstream-upgrade-4-23-dwdrgs9rpkt4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32",
      "name": "f2c90638-d31c-4819-aa00-1f73910cfe32",
      "status": "Failed",
      "startTime": "2026-10-02T23:00:29.412121269Z",
      "endTime": "2026-10-02T23:44:36.589759019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ad2a60>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9h4twv" in resourcegroup="rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx" to finish creating: GET https://rp.j9301120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072",
      "name": "798eb1c5-8054-497c-b466-6bdb523be072",
      "status": "Failed",
      "startTime": "2026-10-02T23:16:07.311984652Z",
      "endTime": "2026-10-02T23:35:13.399526187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ad2a60>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9h4twv" in resourcegroup="rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx" to finish creating: GET https://rp.j9301120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072",
      "name": "798eb1c5-8054-497c-b466-6bdb523be072",
      "status": "Failed",
      "startTime": "2026-10-02T23:16:07.311984652Z",
      "endTime": "2026-10-02T23:35:13.399526187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bc2640>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bc2640>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xc0008d1b20>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-b86qkn" in resourcegroup="rg-np-version-upgrade-b86qkn-wdjflpkjvp8p" to finish creating: GET https://rp.j0034432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24",
      "name": "ba8abfad-4308-4775-a81e-7fccc827da24",
      "status": "Failed",
      "startTime": "2026-10-02T21:59:34.896590709Z",
      "endTime": "2026-10-02T22:18:38.199181809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xc0008d1b20>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-b86qkn" in resourcegroup="rg-np-version-upgrade-b86qkn-wdjflpkjvp8p" to finish creating: GET https://rp.j0034432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24",
      "name": "ba8abfad-4308-4775-a81e-7fccc827da24",
      "status": "Failed",
      "startTime": "2026-10-02T21:59:34.896590709Z",
      "endTime": "2026-10-02T22:18:38.199181809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.006s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-d97ndt" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc001ec6360>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.006s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-d97ndt" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc001ec6360>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001047368>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001047368>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T22:50:00Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0034432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0034432/providers/microsoft.monitor/accounts/services-j0034432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-fdbccb774-c7nq9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0034432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T22:50:00Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0034432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0034432/providers/microsoft.monitor/accounts/services-j0034432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-fdbccb774-c7nq9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0034432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc00022e5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6tx6ckwrt7wg" to finish creating: GET https://rp.j8116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113",
      "name": "2b9f2f0a-a724-431a-8648-3c5581a50113",
      "status": "Failed",
      "startTime": "2026-10-02T21:27:28.841238262Z",
      "endTime": "2026-10-02T21:46:29.997272093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc00022e5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6tx6ckwrt7wg" to finish creating: GET https://rp.j8116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113",
      "name": "2b9f2f0a-a724-431a-8648-3c5581a50113",
      "status": "Failed",
      "startTime": "2026-10-02T21:27:28.841238262Z",
      "endTime": "2026-10-02T21:46:29.997272093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T21:34:19Z
  Ended:       2026-10-02T21:40:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8116608-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00378d080>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-46hf4d98wdxd" to finish creating: GET https://rp.j6618240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "name": "19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "status": "Failed",
      "startTime": "2026-10-02T21:11:48.464360183Z",
      "endTime": "2026-10-02T21:30:51.922878069Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00378d080>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-46hf4d98wdxd" to finish creating: GET https://rp.j6618240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "name": "19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "status": "Failed",
      "startTime": "2026-10-02T21:11:48.464360183Z",
      "endTime": "2026-10-02T21:30:51.922878069Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T20:53:37Z
  Ended:       2026-10-02T20:59:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6618240-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs"
Unexpected error:
    <*fmt.wrapError | 0xc00067a620>: 
    failed to create NodePool np-q47mgs: failed waiting for nodepool="np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs" in resourcegroup="rg-candidate-4-21-8nd2gr7hp9jn" to finish creating: GET https://rp.j6902144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d",
      "name": "fc132130-8033-44d4-826f-921357d1ad9d",
      "status": "Failed",
      "startTime": "2026-10-02T20:44:13.243887943Z",
      "endTime": "2026-10-02T21:03:17.125285435Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs"
Unexpected error:
    <*fmt.wrapError | 0xc00067a620>: 
    failed to create NodePool np-q47mgs: failed waiting for nodepool="np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs" in resourcegroup="rg-candidate-4-21-8nd2gr7hp9jn" to finish creating: GET https://rp.j6902144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d",
      "name": "fc132130-8033-44d4-826f-921357d1ad9d",
      "status": "Failed",
      "startTime": "2026-10-02T20:44:13.243887943Z",
      "endTime": "2026-10-02T21:03:17.125285435Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001e820c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ftx74z7plhzf" to finish creating: GET https://rp.j1487616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "name": "737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "status": "Failed",
      "startTime": "2026-10-02T19:53:28.478249577Z",
      "endTime": "2026-10-02T20:12:35.176603143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001e820c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ftx74z7plhzf" to finish creating: GET https://rp.j1487616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "name": "737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "status": "Failed",
      "startTime": "2026-10-02T19:53:28.478249577Z",
      "endTime": "2026-10-02T20:12:35.176603143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002838820>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002838820>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 40 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T20:04:20Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-rqmvp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-rqmvp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T20:04:20Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-pclwm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-pclwm has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-ffmdk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-ffmdk has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-lqbwl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-lqbwl has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7xqdb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7xqdb has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-xfn2j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-xfn2j has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-hrb67", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-hrb67 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-9rzfn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-9rzfn has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-8rlch", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-8rlch has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-xz6lm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-xz6lm has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-psr9c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-psr9c has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-tdg92", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-tdg92 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nnzlj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nnzlj has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-cp4cg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-cp4cg has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-zs2jk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-zs2jk has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-bft9l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-bft9l has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-l6fg8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-l6fg8 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nkfbd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nkfbd has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-zqdt7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-zqdt7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-6hf78", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-6hf78 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-v2424", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-v2424 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7xlbs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7xlbs has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-gr2nd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-gr2nd has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-hhkrx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-hhkrx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-wnr5j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-wnr5j has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-6kxrp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-6kxrp has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-wxmwp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-wxmwp has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-dgvs2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-dgvs2 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nt5f5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nt5f5 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-qldfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-qldfs has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-vjmrq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-vjmrq has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-mz8lv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-mz8lv has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7r5rf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7r5rf has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-4ljs8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-4ljs8 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-vp8px", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-vp8px has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-t8hh6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-t8hh6 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-9xv2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-9xv2x has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-w4rxn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-w4rxn has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-mfxhq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-mfxhq has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-vngdx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-vngdx has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-c74lpp"
Unexpected error:
    <*fmt.wrapError | 0xc001776360>: 
    DNS for route host agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-c74lpp"
Unexpected error:
    <*fmt.wrapError | 0xc001776360>: 
    DNS for route host agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc00099b300>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8jgnh62lstlc" to finish creating: GET https://rp.j0313216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42",
      "name": "baa8a5de-ef44-437c-8139-ba1877f5af42",
      "status": "Failed",
      "startTime": "2026-10-02T19:34:49.936949596Z",
      "endTime": "2026-10-02T19:53:54.748504782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc00099b300>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8jgnh62lstlc" to finish creating: GET https://rp.j0313216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42",
      "name": "baa8a5de-ef44-437c-8139-ba1877f5af42",
      "status": "Failed",
      "startTime": "2026-10-02T19:34:49.936949596Z",
      "endTime": "2026-10-02T19:53:54.748504782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T19:11:45Z
  Ended:       2026-10-02T19:19:41Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3162112-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000c38480>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-54tj44" in resourcegroup="rg-np-skip-minor-54tj44-rqt5qjd65czw" to finish creating: GET https://rp.j7680896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043",
      "name": "6d49abb2-b87c-4df6-b87c-df15e54da043",
      "status": "Failed",
      "startTime": "2026-10-02T19:28:01.557139687Z",
      "endTime": "2026-10-02T19:47:08.946643419Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000c38480>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-54tj44" in resourcegroup="rg-np-skip-minor-54tj44-rqt5qjd65czw" to finish creating: GET https://rp.j7680896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043",
      "name": "6d49abb2-b87c-4df6-b87c-df15e54da043",
      "status": "Failed",
      "startTime": "2026-10-02T19:28:01.557139687Z",
      "endTime": "2026-10-02T19:47:08.946643419Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T19:07:37Z
  Ended:       2026-10-02T19:15:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7680896-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T18:37:05Z
  Ended:       2026-10-02T18:43:05Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5126912-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:55:55Z
  Ended:       2026-10-02T18:01:55Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4610176-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T19:11:26Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4610176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4610176/providers/microsoft.monitor/accounts/services-j4610176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-69f9b555fd-7q6cb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4610176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T19:11:26Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4610176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4610176/providers/microsoft.monitor/accounts/services-j4610176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-69f9b555fd-7q6cb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4610176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7b9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-92jbnctz6g9n" to finish creating: GET https://rp.j2828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "name": "16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "status": "Failed",
      "startTime": "2026-10-02T17:47:10.869193334Z",
      "endTime": "2026-10-02T18:06:19.043011596Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7b9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-92jbnctz6g9n" to finish creating: GET https://rp.j2828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "name": "16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "status": "Failed",
      "startTime": "2026-10-02T17:47:10.869193334Z",
      "endTime": "2026-10-02T18:06:19.043011596Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 65 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-22dl9", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-22dl9 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dfssq", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dfssq has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-lc7f8", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-lc7f8 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-lrxrn", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-lrxrn has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ws459", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ws459 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-nm4b8", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-nm4b8 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-r88sm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-r88sm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-2f87w", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-2f87w has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-hlcf4", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-hlcf4 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6mnrg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6mnrg has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-t95zp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-t95zp has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jkbjb", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jkbjb has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zcpgg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zcpgg has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-qxvct", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-qxvct has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-mfbkm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-mfbkm has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-mmpwv", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-mmpwv has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-m4v82", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-m4v82 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-fzf98", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-fzf98 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6f2b7", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6f2b7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-z9cp6", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-z9cp6 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-64mk2", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-64mk2 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-glxpz", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-glxpz has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-scxrj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-scxrj has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5876w", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5876w has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-h5skh", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-h5skh has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-77rvv", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-77rvv has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-wxzgc", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-wxzgc has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gxvrp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gxvrp has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-f497d", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-f497d has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zpfft", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zpfft has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-x99lx", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-x99lx has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ksr8r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ksr8r has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-p2k9c", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-p2k9c has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-hp5fb", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-hp5fb has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vjh72", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vjh72 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gfrgk", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gfrgk has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-nrh6c", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-nrh6c has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dcz7s", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dcz7s has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-56n2t", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-56n2t has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-8pwww", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-8pwww has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-sr95r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-sr95r has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-sl2gh", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-sl2gh has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dlbhp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dlbhp has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-pbjdw", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-pbjdw has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-rfjgp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-rfjgp has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-xwfwj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-xwfwj has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gl2xg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gl2xg has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-96lg5", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-96lg5 has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vbcxj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vbcxj has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5sw59", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5sw59 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-l49jm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-l49jm has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jx5v4", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jx5v4 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-s256n", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-s256n has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vnr48", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vnr48 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zrk5n", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zrk5n has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-cc7rs", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-cc7rs has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6j84r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6j84r has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-blmqs", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-blmqs has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-4lnm2", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-4lnm2 has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ppjmp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ppjmp has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-t9nwg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-t9nwg has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-68bhc", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-68bhc has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5fhnt", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5fhnt has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jjrn5", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jjrn5 has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-rzbk6", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-rzbk6 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:55:13Z
  Ended:       2026-10-02T18:02:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2828160-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011448e8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011448e8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.001s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-qjs7pm" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0011ce348>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.001s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-qjs7pm" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0011ce348>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:21:19Z
  Ended:       2026-10-02T17:32:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6990720-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6pa7oq39hmo04tlm81048rosc9slps-ea-listt98lhd", pod="kube-apiserver-594c7bf48-44xbg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6pa7oq39hmo04tlm81048rosc9slps-ea-listt98lhd/kube-apiserver-594c7bf48-44xbg has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:02:52Z
  Ended:       2026-10-02T17:08:49Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7516032-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000ca0a60>: 
    failed to create HCP cluster ea-listpkp2p7: failed waiting for cluster="ea-listpkp2p7" in resourcegroup="ea-list-4ccm8pbgrf47" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "name": "ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "status": "Failed",
      "startTime": "2026-10-02T15:58:05.107845034Z",
      "endTime": "2026-10-02T16:27:08.62997793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 1m14s ago; version 4.21.36 is Partial (want Completed), started 5m36s ago; version 4.21.36 is Partial (want Completed), started 11m4s ago; version 4.21.36 is Partial (want Completed), started 15m6s ago; version 4.21.36 is Partial (want Completed), started 20m46s ago; version 4.21.36 is Partial (want Completed), started 24m30s ago; version 4.21.36 is Partial (want Completed), started 25m39s ago; version 4.21.36 is Partial (want Completed), started 25m51s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000ca0a60>: 
    failed to create HCP cluster ea-listpkp2p7: failed waiting for cluster="ea-listpkp2p7" in resourcegroup="ea-list-4ccm8pbgrf47" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "name": "ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "status": "Failed",
      "startTime": "2026-10-02T15:58:05.107845034Z",
      "endTime": "2026-10-02T16:27:08.62997793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 1m14s ago; version 4.21.36 is Partial (want Completed), started 5m36s ago; version 4.21.36 is Partial (want Completed), started 11m4s ago; version 4.21.36 is Partial (want Completed), started 15m6s ago; version 4.21.36 is Partial (want Completed), started 20m46s ago; version 4.21.36 is Partial (want Completed), started 24m30s ago; version 4.21.36 is Partial (want Completed), started 25m39s ago; version 4.21.36 is Partial (want Completed), started 25m51s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014427a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-h248xrjjcrlg" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "name": "38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "status": "Failed",
      "startTime": "2026-10-02T16:37:00.371020776Z",
      "endTime": "2026-10-02T16:56:08.531132296Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014427a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-h248xrjjcrlg" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "name": "38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "status": "Failed",
      "startTime": "2026-10-02T16:37:00.371020776Z",
      "endTime": "2026-10-02T16:56:08.531132296Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bdc160>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bdc160>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T16:13:37Z
  Ended:       2026-10-02T16:20:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4233600-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T16:10:23Z
  Ended:       2026-10-02T16:17:22Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7055872-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc0011632a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-sq4gl58phslg" to finish creating: GET https://rp.j7228672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "name": "1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "status": "Failed",
      "startTime": "2026-10-02T16:32:42.711328508Z",
      "endTime": "2026-10-02T16:51:42.25861049Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc0011632a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-sq4gl58phslg" to finish creating: GET https://rp.j7228672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "name": "1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "status": "Failed",
      "startTime": "2026-10-02T16:32:42.711328508Z",
      "endTime": "2026-10-02T16:51:42.25861049Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml"
Unexpected error:
    <*fmt.wrapError | 0xc000c8a7e0>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml" in resourcegroup="rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx" to finish creating: GET https://rp.j9111424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377",
      "name": "37ab050f-9ade-44b7-9470-1944f6bc1377",
      "status": "Failed",
      "startTime": "2026-10-02T16:08:46.628951449Z",
      "endTime": "2026-10-02T16:27:53.982628223Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml"
Unexpected error:
    <*fmt.wrapError | 0xc000c8a7e0>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml" in resourcegroup="rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx" to finish creating: GET https://rp.j9111424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377",
      "name": "37ab050f-9ade-44b7-9470-1944f6bc1377",
      "status": "Failed",
      "startTime": "2026-10-02T16:08:46.628951449Z",
      "endTime": "2026-10-02T16:27:53.982628223Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T15:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-k7gkp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-k7gkp has been in a non-ready state for longer than 5 minutes.

alert fired 14 time(s) (2 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T16:07:06Z
  Ended:       2026-10-02T16:28:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6ocek7h2b81l310othj19g5cs2hvdm-h6f8s9c6d5a8t7c", pod="router-7466989cdd-4xgts", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6ocek7h2b81l310othj19g5cs2hvdm-h6f8s9c6d5a8t7c/router-7466989cdd-4xgts has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T16:07:07Z
  Ended:       2026-10-02T16:28:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6oclvhiuo0ben6245b5rc60j4ul3v1-v20261001", pod="router-58b59f844f-6fgw8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6oclvhiuo0ben6245b5rc60j4ul3v1-v20261001/router-58b59f844f-6fgw8 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs"
Unexpected error:
    <*fmt.wrapError | 0xc000bb3a80>: 
    failed to create NodePool np-xbxlzs: failed waiting for nodepool="np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs" in resourcegroup="rg-candidate-4-20-gwcqh9m2pngx" to finish creating: GET https://rp.j2125184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971",
      "name": "2e698a08-7d53-4eac-b66e-e0b997094971",
      "status": "Failed",
      "startTime": "2026-10-02T15:16:45.725296348Z",
      "endTime": "2026-10-02T15:35:49.304545041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs"
Unexpected error:
    <*fmt.wrapError | 0xc000bb3a80>: 
    failed to create NodePool np-xbxlzs: failed waiting for nodepool="np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs" in resourcegroup="rg-candidate-4-20-gwcqh9m2pngx" to finish creating: GET https://rp.j2125184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971",
      "name": "2e698a08-7d53-4eac-b66e-e0b997094971",
      "status": "Failed",
      "startTime": "2026-10-02T15:16:45.725296348Z",
      "endTime": "2026-10-02T15:35:49.304545041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T15:21:16Z
  Ended:       2026-10-02T15:28:15Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2125184-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T15:09:09Z
  Ended:       2026-10-02T15:15:04Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6816000-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T15:01:55Z
  Ended:       2026-10-02T15:07:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9080064-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T13:57:08Z
  Ended:       2026-10-02T14:15:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4702592-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00156e168>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00156e168>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000a14dc0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-4jgj2xjf9rbf" to finish creating: GET https://rp.j6116992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826",
      "name": "c833939d-30b9-4409-82fd-33d2b22e8826",
      "status": "Failed",
      "startTime": "2026-10-02T13:10:20.783755056Z",
      "endTime": "2026-10-02T13:29:28.266052933Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000a14dc0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-4jgj2xjf9rbf" to finish creating: GET https://rp.j6116992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826",
      "name": "c833939d-30b9-4409-82fd-33d2b22e8826",
      "status": "Failed",
      "startTime": "2026-10-02T13:10:20.783755056Z",
      "endTime": "2026-10-02T13:29:28.266052933Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000c23e80>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.004s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000c23e80>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.004s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0028843f0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kqrgzxwnxwgd" to finish creating: GET https://rp.j8271104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "name": "7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "status": "Failed",
      "startTime": "2026-10-02T12:59:29.315762309Z",
      "endTime": "2026-10-02T13:18:36.006994146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-597b9-prdsc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-597b9-prdsc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0028843f0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kqrgzxwnxwgd" to finish creating: GET https://rp.j8271104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "name": "7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "status": "Failed",
      "startTime": "2026-10-02T12:59:29.315762309Z",
      "endTime": "2026-10-02T13:18:36.006994146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-597b9-prdsc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-597b9-prdsc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:212]: Timed out after 2700.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:210 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:212]: Timed out after 2700.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:210 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gc94n755ktzv/cluster-candidate-5-0-l6z5kw should provision
Unexpected error:
    <*fmt.wrapError | 0xc001032d80>: 
    failed to create HCP cluster cluster-candidate-5-0-l6z5kw: failed waiting for cluster="cluster-candidate-5-0-l6z5kw" in resourcegroup="rg-candidate-5-0-gc94n755ktzv" to finish creating: GET https://rp.j1697920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "name": "af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "status": "Failed",
      "startTime": "2026-10-02T11:49:05.164042474Z",
      "endTime": "2026-10-02T12:18:11.434067357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gc94n755ktzv/cluster-candidate-5-0-l6z5kw should provision
Unexpected error:
    <*fmt.wrapError | 0xc001032d80>: 
    failed to create HCP cluster cluster-candidate-5-0-l6z5kw: failed waiting for cluster="cluster-candidate-5-0-l6z5kw" in resourcegroup="rg-candidate-5-0-gc94n755ktzv" to finish creating: GET https://rp.j1697920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "name": "af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "status": "Failed",
      "startTime": "2026-10-02T11:49:05.164042474Z",
      "endTime": "2026-10-02T12:18:11.434067357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:19:46Z
  Ended:       2026-10-02T12:37:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T12:19:46Z
  Ended:       2026-10-02T12:37:46Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:24:46Z
  Ended:       2026-10-02T12:39:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:11:46Z
  Ended:       2026-10-02T12:41:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3ec768d5-6cf8-4b9c-a960-342894bf7da7"
  Description: Pod ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-02T12:11:46Z
  Ended:       2026-10-02T12:41:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3ec768d5-6cf8-4b9c-a960-342894bf7da7"
  Description: Pod ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010e1940>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-w27q7sgvxjfv" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699",
      "name": "d3cd18ff-072e-464f-89ca-f4c27910d699",
      "status": "Failed",
      "startTime": "2026-10-02T11:50:03.491486549Z",
      "endTime": "2026-10-02T12:09:10.243715073Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010e1940>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-w27q7sgvxjfv" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699",
      "name": "d3cd18ff-072e-464f-89ca-f4c27910d699",
      "status": "Failed",
      "startTime": "2026-10-02T11:50:03.491486549Z",
      "endTime": "2026-10-02T12:09:10.243715073Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000f670c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-tjrqglfwz7jf" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "name": "92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "status": "Failed",
      "startTime": "2026-10-02T11:51:54.872760103Z",
      "endTime": "2026-10-02T12:11:00.580352219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine acr-pull-create-np-1-rg8t6-jcb2q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine acr-pull-create-np-1-rg8t6-jcb2q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000f670c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-tjrqglfwz7jf" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "name": "92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "status": "Failed",
      "startTime": "2026-10-02T11:51:54.872760103Z",
      "endTime": "2026-10-02T12:11:00.580352219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine acr-pull-create-np-1-rg8t6-jcb2q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine acr-pull-create-np-1-rg8t6-jcb2q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T11:59:15Z
  Ended:       2026-10-02T12:14:18Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0488832-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002032960>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-s47ln9cbqhqp" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "name": "b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "status": "Failed",
      "startTime": "2026-10-02T11:09:22.074259734Z",
      "endTime": "2026-10-02T11:28:22.963531957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002032960>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-s47ln9cbqhqp" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "name": "b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "status": "Failed",
      "startTime": "2026-10-02T11:09:22.074259734Z",
      "endTime": "2026-10-02T11:28:22.963531957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.009s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-dg8wqw" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0015c3050>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.009s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-dg8wqw" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0015c3050>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000dc15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000dc15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00063e600>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-snc8bz4l2lpm" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999",
      "name": "377e91a7-9fb9-4364-a4c3-94d301101999",
      "status": "Failed",
      "startTime": "2026-10-02T11:41:21.075539741Z",
      "endTime": "2026-10-02T12:00:23.993693871Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00063e600>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-snc8bz4l2lpm" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999",
      "name": "377e91a7-9fb9-4364-a4c3-94d301101999",
      "status": "Failed",
      "startTime": "2026-10-02T11:41:21.075539741Z",
      "endTime": "2026-10-02T12:00:23.993693871Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:193]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015c66c0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nqllmtrxwh84" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-nqllmtrxwh84, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:193]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015c66c0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nqllmtrxwh84" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-nqllmtrxwh84, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:416]: failed to poll cluster "cidr-connectivity-test" authorized CIDRs update to completion (timeout '10.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:416]: failed to poll cluster "cidr-connectivity-test" authorized CIDRs update to completion (timeout '10.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9"
Unexpected error:
    <*fmt.wrapError | 0xc001238cc0>: 
    failed to create NodePool np-j4v4c9: failed waiting for nodepool="np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9" in resourcegroup="rg-candidate-4-20-ck8mc4szhlcn" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "name": "f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "status": "Failed",
      "startTime": "2026-10-02T11:08:30.228624897Z",
      "endTime": "2026-10-02T11:27:32.877879167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9"
Unexpected error:
    <*fmt.wrapError | 0xc001238cc0>: 
    failed to create NodePool np-j4v4c9: failed waiting for nodepool="np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9" in resourcegroup="rg-candidate-4-20-ck8mc4szhlcn" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "name": "f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "status": "Failed",
      "startTime": "2026-10-02T11:08:30.228624897Z",
      "endTime": "2026-10-02T11:27:32.877879167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d8120>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-9h5ws9hlzq4p" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-9h5ws9hlzq4p, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d8120>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-9h5ws9hlzq4p" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-9h5ws9hlzq4p, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*fmt.wrapErrors | 0xc0013d63c0>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d4f6lxz4s2dz" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-d4f6lxz4s2dz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*fmt.wrapErrors | 0xc0013d63c0>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d4f6lxz4s2dz" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-d4f6lxz4s2dz, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:369]: failed to update containerRegistry via PATCH
Unexpected error:
    <*fmt.wrapErrors | 0xc000fa8690>: 
    failed waiting for hcpCluster="acr-pull-create" in resourcegroup="acr-pull-create-ljhpblch2468" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20261001 for cluster acr-pull-create in resource group acr-pull-create-ljhpblch2468, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:369]: failed to update containerRegistry via PATCH
Unexpected error:
    <*fmt.wrapErrors | 0xc000fa8690>: 
    failed waiting for hcpCluster="acr-pull-create" in resourcegroup="acr-pull-create-ljhpblch2468" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20261001 for cluster acr-pull-create in resource group acr-pull-create-ljhpblch2468, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T11:16:17Z
  Ended:       2026-10-02T11:26:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7658624-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6k02kauka61vjc7pvhhdhndnsmtknd-ea-list6nlmtx", pod="kube-apiserver-67bb546754-b454t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6k02kauka61vjc7pvhhdhndnsmtknd-ea-list6nlmtx/kube-apiserver-67bb546754-b454t has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T09:12:36Z
  Ended:       2026-10-02T09:27:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9667072-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T09:12:36Z
  Ended:       2026-10-02T09:30:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9667072-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-10-02T09:12:36Z
  Ended:       2026-10-02T09:28:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9667072-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000b84f00>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2vzgl45q9ll4" to finish creating: GET https://rp.j0616704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "name": "992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "status": "Failed",
      "startTime": "2026-10-02T08:27:53.917230714Z",
      "endTime": "2026-10-02T08:46:59.056787427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000b84f00>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2vzgl45q9ll4" to finish creating: GET https://rp.j0616704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "name": "992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "status": "Failed",
      "startTime": "2026-10-02T08:27:53.917230714Z",
      "endTime": "2026-10-02T08:46:59.056787427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T08:10:58Z
  Ended:       2026-10-02T08:29:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0616704-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh"
Unexpected error:
    <*fmt.wrapError | 0xc000eb30a0>: 
    failed to create NodePool np-7ctmvh: failed waiting for nodepool="np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh" in resourcegroup="rg-candidate-4-23-ztlck247gbc7" to finish creating: GET https://rp.j6550784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482",
      "name": "28de2827-3d1e-473f-9ac1-7443442e5482",
      "status": "Failed",
      "startTime": "2026-10-02T06:54:46.375852677Z",
      "endTime": "2026-10-02T07:13:49.710009064Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh"
Unexpected error:
    <*fmt.wrapError | 0xc000eb30a0>: 
    failed to create NodePool np-7ctmvh: failed waiting for nodepool="np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh" in resourcegroup="rg-candidate-4-23-ztlck247gbc7" to finish creating: GET https://rp.j6550784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482",
      "name": "28de2827-3d1e-473f-9ac1-7443442e5482",
      "status": "Failed",
      "startTime": "2026-10-02T06:54:46.375852677Z",
      "endTime": "2026-10-02T07:13:49.710009064Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-10-02T04:18:24.964Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-02T04:18:25.604Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:3a6b5dce-c503-483f-868b-3656736645ed cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j6706176.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j6706176-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j6706176-1 secret:hcp-underlay-j6706176-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-02T04:18:25.604Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:18:26.032Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-02T04:18:26.347Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-02T04:18:26.347Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:18:26.470Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:23:32.794Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:23:32.795Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000e9d1e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-z847fd" in resourcegroup="rg-np-version-upgrade-z847fd-9d7c47wqwcb4" to finish creating: GET https://rp.j1887616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "name": "6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "status": "Failed",
      "startTime": "2026-10-02T04:46:32.427711572Z",
      "endTime": "2026-10-02T05:05:35.48212253Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000e9d1e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-z847fd" in resourcegroup="rg-np-version-upgrade-z847fd-9d7c47wqwcb4" to finish creating: GET https://rp.j1887616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "name": "6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "status": "Failed",
      "startTime": "2026-10-02T04:46:32.427711572Z",
      "endTime": "2026-10-02T05:05:35.48212253Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-10-02T04:04:12.931Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-02T04:04:13.290Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:ad930020-a0d3-4a84-b682-06665523a404 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j3304448.canadacentral-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j3304448-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j3304448-1 secret:hcp-underlay-j3304448-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-02T04:04:13.290Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:04:13.437Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-02T04:04:13.594Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-02T04:04:13.594Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:04:13.638Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:09:17.935Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:09:17.936Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-wzq9v", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-wzq9v has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-qj4ln", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-qj4ln has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-w44fw", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-w44fw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-66rf9", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-66rf9 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-4p6tc", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-4p6tc has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000f90e60>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-gxq8mc" in resourcegroup="rg-np-skip-minor-gxq8mc-kwv67gxfnh4r" to finish creating: GET https://rp.j3775744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "name": "5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "status": "Failed",
      "startTime": "2026-10-02T04:52:30.925064823Z",
      "endTime": "2026-10-02T05:11:35.851064932Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000f90e60>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-gxq8mc" in resourcegroup="rg-np-skip-minor-gxq8mc-kwv67gxfnh4r" to finish creating: GET https://rp.j3775744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "name": "5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "status": "Failed",
      "startTime": "2026-10-02T04:52:30.925064823Z",
      "endTime": "2026-10-02T05:11:35.851064932Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000fef1c0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-svbr9q" in resourcegroup="rg-np-downgrade-svbr9q-kvqmb7zbtpsq" to finish creating: GET https://rp.j3810048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "name": "556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "status": "Failed",
      "startTime": "2026-10-02T01:38:44.187977267Z",
      "endTime": "2026-10-02T01:57:50.438850256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000fef1c0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-svbr9q" in resourcegroup="rg-np-downgrade-svbr9q-kvqmb7zbtpsq" to finish creating: GET https://rp.j3810048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "name": "556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "status": "Failed",
      "startTime": "2026-10-02T01:38:44.187977267Z",
      "endTime": "2026-10-02T01:57:50.438850256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jcbqlv" for cluster "cluster-candidate-4-20-jcbqlv"
Unexpected error:
    <*fmt.wrapError | 0xc002b94f00>: 
    failed to create NodePool np-jcbqlv: failed waiting for nodepool="np-jcbqlv" for cluster "cluster-candidate-4-20-jcbqlv" in resourcegroup="rg-candidate-4-20-lvv7b2lssbpl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84cb018b-4347-4f56-b1c7-88d7604492f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84cb018b-4347-4f56-b1c7-88d7604492f9",
      "name": "84cb018b-4347-4f56-b1c7-88d7604492f9",
      "status": "Failed",
      "startTime": "2026-10-03T00:32:09.437716177Z",
      "endTime": "2026-10-03T00:51:11.761764992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lvv7b2lssbpl-candidate-jcbqlv--managed/providers/Microsoft.Compute/virtualMachines/c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lvv7b2lssbpl-candidate-jcbqlv--managed/providers/Microsoft.Compute/virtualMachines/c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jcbqlv" for cluster "cluster-candidate-4-20-jcbqlv"
Unexpected error:
    <*fmt.wrapError | 0xc002b94f00>: 
    failed to create NodePool np-jcbqlv: failed waiting for nodepool="np-jcbqlv" for cluster "cluster-candidate-4-20-jcbqlv" in resourcegroup="rg-candidate-4-20-lvv7b2lssbpl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84cb018b-4347-4f56-b1c7-88d7604492f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84cb018b-4347-4f56-b1c7-88d7604492f9",
      "name": "84cb018b-4347-4f56-b1c7-88d7604492f9",
      "status": "Failed",
      "startTime": "2026-10-03T00:32:09.437716177Z",
      "endTime": "2026-10-03T00:51:11.761764992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lvv7b2lssbpl-candidate-jcbqlv--managed/providers/Microsoft.Compute/virtualMachines/c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-lvv7b2lssbpl-candidate-jcbqlv--managed/providers/Microsoft.Compute/virtualMachines/c1t3y9h8f3v9w7t-np-jcbqlv-zlpbm-ww6xn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00113e680>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-9tnt4zgp45ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84e6a04c-39e2-4627-85f5-80cd1669879f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84e6a04c-39e2-4627-85f5-80cd1669879f",
      "name": "84e6a04c-39e2-4627-85f5-80cd1669879f",
      "status": "Failed",
      "startTime": "2026-10-02T23:58:26.609352403Z",
      "endTime": "2026-10-03T00:17:31.644821105Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-mmbgh-jld78: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9tnt4zgp45ww--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-mmbgh-jld78 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-mmbgh-jld78: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9tnt4zgp45ww--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-mmbgh-jld78 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00113e680>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-9tnt4zgp45ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84e6a04c-39e2-4627-85f5-80cd1669879f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/84e6a04c-39e2-4627-85f5-80cd1669879f",
      "name": "84e6a04c-39e2-4627-85f5-80cd1669879f",
      "status": "Failed",
      "startTime": "2026-10-02T23:58:26.609352403Z",
      "endTime": "2026-10-03T00:17:31.644821105Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-mmbgh-jld78: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9tnt4zgp45ww--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-mmbgh-jld78 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-mmbgh-jld78: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9tnt4zgp45ww--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-mmbgh-jld78 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:286]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000a814a0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:286]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000a814a0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-kf8zjf" to 4.21.0-0.nightly-multi-2026-10-02-021649
Unexpected error:
    <*exported.ResponseError | 0xc000d32ae0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-9wcjt7cq5sl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-kf8zjf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-kf8zjf" to 4.21.0-0.nightly-multi-2026-10-02-021649
Unexpected error:
    <*exported.ResponseError | 0xc000d32ae0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-9wcjt7cq5sl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-kf8zjf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000cdbad0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-hbvfkpmwjx97/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000cdbad0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-hbvfkpmwjx97/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-s49xkm" for cluster "cluster-nightly-4-20-s49xkm"
Unexpected error:
    <*fmt.wrapError | 0xc000ad8640>: 
    failed to create NodePool np-s49xkm: failed starting nodepool creation "np-s49xkm" for cluster "cluster-nightly-4-20-s49xkm" in resourcegroup="rg-nightly-4-20-mhnxwl8ndm9g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-mhnxwl8ndm9g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-s49xkm/nodePools/np-s49xkm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-s49xkm" for cluster "cluster-nightly-4-20-s49xkm"
Unexpected error:
    <*fmt.wrapError | 0xc000ad8640>: 
    failed to create NodePool np-s49xkm: failed starting nodepool creation "np-s49xkm" for cluster "cluster-nightly-4-20-s49xkm" in resourcegroup="rg-nightly-4-20-mhnxwl8ndm9g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-mhnxwl8ndm9g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-s49xkm/nodePools/np-s49xkm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0023c62d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-w5hzr8g77c9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0023c62d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-w5hzr8g77c9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc0012cc100>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-qmt425cc9vkv": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qmt425cc9vkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc0012cc100>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-qmt425cc9vkv": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qmt425cc9vkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0xc0006110a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-bh8wcncrztgb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-bh8wcncrztgb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0xc0006110a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-bh8wcncrztgb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-bh8wcncrztgb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-pwn8nb at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000692a40>: 
    failed to create HCP cluster np-skip-minor-pwn8nb: failed waiting for cluster="np-skip-minor-pwn8nb" in resourcegroup="rg-np-skip-minor-pwn8nb-fcqmqdxvjmqx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9282a0a5-6015-4ac4-8987-2148df07fa63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9282a0a5-6015-4ac4-8987-2148df07fa63",
      "name": "9282a0a5-6015-4ac4-8987-2148df07fa63",
      "status": "Failed",
      "startTime": "2026-10-03T00:07:14.845401374Z",
      "endTime": "2026-10-03T00:36:15.664777437Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-pwn8nb at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000692a40>: 
    failed to create HCP cluster np-skip-minor-pwn8nb: failed waiting for cluster="np-skip-minor-pwn8nb" in resourcegroup="rg-np-skip-minor-pwn8nb-fcqmqdxvjmqx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9282a0a5-6015-4ac4-8987-2148df07fa63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9282a0a5-6015-4ac4-8987-2148df07fa63",
      "name": "9282a0a5-6015-4ac4-8987-2148df07fa63",
      "status": "Failed",
      "startTime": "2026-10-03T00:07:14.845401374Z",
      "endTime": "2026-10-03T00:36:15.664777437Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-21-8rfqwl" is viable
Unexpected error:
    <*errors.joinError | 0xc00127afc0>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-21-8rfqwl" is viable
Unexpected error:
    <*errors.joinError | 0xc00127afc0>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-fts22d" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc00111ac60>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-fts22d: failed waiting for cluster="cp-ystream-upgrade-5-0-fts22d" in resourcegroup="rg-cp-ystream-upgrade-5-0-mljdp8jfhscw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ec54c6-385b-4f23-8fbf-a1f041a3140c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ec54c6-385b-4f23-8fbf-a1f041a3140c",
      "name": "51ec54c6-385b-4f23-8fbf-a1f041a3140c",
      "status": "Failed",
      "startTime": "2026-10-02T23:43:28.788659857Z",
      "endTime": "2026-10-03T00:12:29.488557875Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-fts22d" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc00111ac60>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-fts22d: failed waiting for cluster="cp-ystream-upgrade-5-0-fts22d" in resourcegroup="rg-cp-ystream-upgrade-5-0-mljdp8jfhscw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ec54c6-385b-4f23-8fbf-a1f041a3140c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51ec54c6-385b-4f23-8fbf-a1f041a3140c",
      "name": "51ec54c6-385b-4f23-8fbf-a1f041a3140c",
      "status": "Failed",
      "startTime": "2026-10-02T23:43:28.788659857Z",
      "endTime": "2026-10-03T00:12:29.488557875Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015c28d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-czq6qgmpwn7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015c28d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-czq6qgmpwn7z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-d88bcx" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0xc0010032f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bh6j7gtlcdf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-d88bcx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-d88bcx" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0xc0010032f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bh6j7gtlcdf2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-d88bcx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-m2xdl7kgwht2/cluster-nightly-4-22-q779x6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009c8f00>: 
    failed to create HCP cluster cluster-nightly-4-22-q779x6: failed waiting for cluster="cluster-nightly-4-22-q779x6" in resourcegroup="rg-nightly-4-22-m2xdl7kgwht2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecf6db04-6bc9-42e0-9339-05a023c3c7c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecf6db04-6bc9-42e0-9339-05a023c3c7c2",
      "name": "ecf6db04-6bc9-42e0-9339-05a023c3c7c2",
      "status": "Failed",
      "startTime": "2026-10-02T23:39:38.855575535Z",
      "endTime": "2026-10-03T00:08:39.476306937Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-m2xdl7kgwht2/cluster-nightly-4-22-q779x6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009c8f00>: 
    failed to create HCP cluster cluster-nightly-4-22-q779x6: failed waiting for cluster="cluster-nightly-4-22-q779x6" in resourcegroup="rg-nightly-4-22-m2xdl7kgwht2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecf6db04-6bc9-42e0-9339-05a023c3c7c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecf6db04-6bc9-42e0-9339-05a023c3c7c2",
      "name": "ecf6db04-6bc9-42e0-9339-05a023c3c7c2",
      "status": "Failed",
      "startTime": "2026-10-02T23:39:38.855575535Z",
      "endTime": "2026-10-03T00:08:39.476306937Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d72ea0>: 
    failed to create HCP cluster np-dg-minor-x975lx: failed waiting for cluster="np-dg-minor-x975lx" in resourcegroup="rg-np-dg-minor-x975lx-snvpx8zstwzg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bf1cc23-284f-4b81-90c6-b740356d67dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bf1cc23-284f-4b81-90c6-b740356d67dd",
      "name": "1bf1cc23-284f-4b81-90c6-b740356d67dd",
      "status": "Failed",
      "startTime": "2026-10-03T00:09:01.246824643Z",
      "endTime": "2026-10-03T00:38:05.670946046Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d72ea0>: 
    failed to create HCP cluster np-dg-minor-x975lx: failed waiting for cluster="np-dg-minor-x975lx" in resourcegroup="rg-np-dg-minor-x975lx-snvpx8zstwzg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bf1cc23-284f-4b81-90c6-b740356d67dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bf1cc23-284f-4b81-90c6-b740356d67dd",
      "name": "1bf1cc23-284f-4b81-90c6-b740356d67dd",
      "status": "Failed",
      "startTime": "2026-10-03T00:09:01.246824643Z",
      "endTime": "2026-10-03T00:38:05.670946046Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-qssg82" to 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*exported.ResponseError | 0xc0014d0480>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-cz4jzkrzbs6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-qssg82
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-qssg82" to 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*exported.ResponseError | 0xc0014d0480>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-cz4jzkrzbs6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-qssg82
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0026fb2f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cr2tzrn9kjll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0026fb2f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cr2tzrn9kjll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc000de8120>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-4slk4t" in resourcegroup="rg-np-version-upgrade-4slk4t-r4jcn6z5vwbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-4slk4t-r4jcn6z5vwbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-4slk4t/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc000de8120>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-4slk4t" in resourcegroup="rg-np-version-upgrade-4slk4t-r4jcn6z5vwbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-4slk4t-r4jcn6z5vwbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-4slk4t/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc001074300>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vhhr845dxfj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc001074300>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vhhr845dxfj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-wlr9js" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000d5cb20>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-wlr9js: failed waiting for cluster="cp-ystream-upgrade-4-23-wlr9js" in resourcegroup="rg-cp-ystream-upgrade-4-23-hsrbsrxk7qbw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a34556-9c6b-44f5-9047-c18c0b099bd1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a34556-9c6b-44f5-9047-c18c0b099bd1",
      "name": "c9a34556-9c6b-44f5-9047-c18c0b099bd1",
      "status": "Failed",
      "startTime": "2026-10-02T23:42:52.038313719Z",
      "endTime": "2026-10-03T00:11:59.486058349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-wlr9js" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000d5cb20>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-wlr9js: failed waiting for cluster="cp-ystream-upgrade-4-23-wlr9js" in resourcegroup="rg-cp-ystream-upgrade-4-23-hsrbsrxk7qbw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a34556-9c6b-44f5-9047-c18c0b099bd1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a34556-9c6b-44f5-9047-c18c0b099bd1",
      "name": "c9a34556-9c6b-44f5-9047-c18c0b099bd1",
      "status": "Failed",
      "startTime": "2026-10-02T23:42:52.038313719Z",
      "endTime": "2026-10-03T00:11:59.486058349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0022ce920>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-fgvr8lf6bszk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/db510478-c50d-4f9a-86b6-3bb190ec8daa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/db510478-c50d-4f9a-86b6-3bb190ec8daa",
      "name": "db510478-c50d-4f9a-86b6-3bb190ec8daa",
      "status": "Failed",
      "startTime": "2026-10-02T17:45:05.919871003Z",
      "endTime": "2026-10-02T18:14:10.848146933Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0022ce920>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-fgvr8lf6bszk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/db510478-c50d-4f9a-86b6-3bb190ec8daa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/db510478-c50d-4f9a-86b6-3bb190ec8daa",
      "name": "db510478-c50d-4f9a-86b6-3bb190ec8daa",
      "status": "Failed",
      "startTime": "2026-10-02T17:45:05.919871003Z",
      "endTime": "2026-10-02T18:14:10.848146933Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001361080>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001361080>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc001253878>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node c7z4o0b2f4w6f5y-np-1-7fd77-f8d28: failed to get pod logs for pod fips-check-z86wg in namespace e2e-fips-test-pdk46 on node c7z4o0b2f4w6f5y-np-1-7fd77-f8d28 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-z86wg)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc001253878>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node c7z4o0b2f4w6f5y-np-1-7fd77-f8d28: failed to get pod logs for pod fips-check-z86wg in namespace e2e-fips-test-pdk46 on node c7z4o0b2f4w6f5y-np-1-7fd77-f8d28 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-z86wg)
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000c93500>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-tlcz46v8f5nd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9d16ce0e-bad0-4102-85e4-8ff744b9a2fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9d16ce0e-bad0-4102-85e4-8ff744b9a2fd",
      "name": "9d16ce0e-bad0-4102-85e4-8ff744b9a2fd",
      "status": "Failed",
      "startTime": "2026-10-02T19:19:55.216406379Z",
      "endTime": "2026-10-02T19:39:00.624366265Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-tlcz46v8f5nd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-tlcz46v8f5nd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000c93500>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-tlcz46v8f5nd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9d16ce0e-bad0-4102-85e4-8ff744b9a2fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9d16ce0e-bad0-4102-85e4-8ff744b9a2fd",
      "name": "9d16ce0e-bad0-4102-85e4-8ff744b9a2fd",
      "status": "Failed",
      "startTime": "2026-10-02T19:19:55.216406379Z",
      "endTime": "2026-10-02T19:39:00.624366265Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-tlcz46v8f5nd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-tlcz46v8f5nd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-9wdrv-tkpqf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc00109c260>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-vl5bmxfjpvnl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/76a0d1d5-c260-480b-9486-2626418cd0aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/76a0d1d5-c260-480b-9486-2626418cd0aa",
      "name": "76a0d1d5-c260-480b-9486-2626418cd0aa",
      "status": "Failed",
      "startTime": "2026-10-02T15:16:07.904782817Z",
      "endTime": "2026-10-02T15:35:14.636915839Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5k5p2r5e7o0k9d-np-1-pmf67-7nch9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vl5bmxfjpvnl--managed/providers/Microsoft.Compute/virtualMachines/p5k5p2r5e7o0k9d-np-1-pmf67-7nch9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5k5p2r5e7o0k9d-np-1-pmf67-7nch9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vl5bmxfjpvnl--managed/providers/Microsoft.Compute/virtualMachines/p5k5p2r5e7o0k9d-np-1-pmf67-7nch9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc00109c260>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-vl5bmxfjpvnl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/76a0d1d5-c260-480b-9486-2626418cd0aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/76a0d1d5-c260-480b-9486-2626418cd0aa",
      "name": "76a0d1d5-c260-480b-9486-2626418cd0aa",
      "status": "Failed",
      "startTime": "2026-10-02T15:16:07.904782817Z",
      "endTime": "2026-10-02T15:35:14.636915839Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5k5p2r5e7o0k9d-np-1-pmf67-7nch9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vl5bmxfjpvnl--managed/providers/Microsoft.Compute/virtualMachines/p5k5p2r5e7o0k9d-np-1-pmf67-7nch9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5k5p2r5e7o0k9d-np-1-pmf67-7nch9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vl5bmxfjpvnl--managed/providers/Microsoft.Compute/virtualMachines/p5k5p2r5e7o0k9d-np-1-pmf67-7nch9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qx4psk" for cluster "cluster-candidate-5-1-qx4psk"
Unexpected error:
    <*fmt.wrapError | 0xc0013d5b40>: 
    failed to create NodePool np-qx4psk: failed waiting for nodepool="np-qx4psk" for cluster "cluster-candidate-5-1-qx4psk" in resourcegroup="rg-candidate-5-1-kt5m7922qfx8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/08a3ce96-4944-47f9-8702-e2ad9f78a6cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/08a3ce96-4944-47f9-8702-e2ad9f78a6cf",
      "name": "08a3ce96-4944-47f9-8702-e2ad9f78a6cf",
      "status": "Failed",
      "startTime": "2026-10-02T15:30:38.36539796Z",
      "endTime": "2026-10-02T15:49:40.880753059Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kt5m7922qfx8-candidate-qx4psk--managed/providers/Microsoft.Compute/virtualMachines/z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kt5m7922qfx8-candidate-qx4psk--managed/providers/Microsoft.Compute/virtualMachines/z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qx4psk" for cluster "cluster-candidate-5-1-qx4psk"
Unexpected error:
    <*fmt.wrapError | 0xc0013d5b40>: 
    failed to create NodePool np-qx4psk: failed waiting for nodepool="np-qx4psk" for cluster "cluster-candidate-5-1-qx4psk" in resourcegroup="rg-candidate-5-1-kt5m7922qfx8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/08a3ce96-4944-47f9-8702-e2ad9f78a6cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/08a3ce96-4944-47f9-8702-e2ad9f78a6cf",
      "name": "08a3ce96-4944-47f9-8702-e2ad9f78a6cf",
      "status": "Failed",
      "startTime": "2026-10-02T15:30:38.36539796Z",
      "endTime": "2026-10-02T15:49:40.880753059Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kt5m7922qfx8-candidate-qx4psk--managed/providers/Microsoft.Compute/virtualMachines/z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-kt5m7922qfx8-candidate-qx4psk--managed/providers/Microsoft.Compute/virtualMachines/z1v0n8y8q7v5r0b-np-qx4psk-v8zlj-7bpw8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-494d7tjl9z6l/cluster-candidate-4-20-zdwtn5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c31000>: 
    failed to create HCP cluster cluster-candidate-4-20-zdwtn5: failed waiting for cluster="cluster-candidate-4-20-zdwtn5" in resourcegroup="rg-candidate-4-20-494d7tjl9z6l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/cecb851f-dab3-443d-be30-9e210495bb30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/cecb851f-dab3-443d-be30-9e210495bb30",
      "name": "cecb851f-dab3-443d-be30-9e210495bb30",
      "status": "Failed",
      "startTime": "2026-10-02T14:47:02.598518687Z",
      "endTime": "2026-10-02T15:16:11.481682438Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-494d7tjl9z6l/cluster-candidate-4-20-zdwtn5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c31000>: 
    failed to create HCP cluster cluster-candidate-4-20-zdwtn5: failed waiting for cluster="cluster-candidate-4-20-zdwtn5" in resourcegroup="rg-candidate-4-20-494d7tjl9z6l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/cecb851f-dab3-443d-be30-9e210495bb30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/cecb851f-dab3-443d-be30-9e210495bb30",
      "name": "cecb851f-dab3-443d-be30-9e210495bb30",
      "status": "Failed",
      "startTime": "2026-10-02T14:47:02.598518687Z",
      "endTime": "2026-10-02T15:16:11.481682438Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc002486980>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4z7j926fljlc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f9754b3c-7f81-4d15-8726-14659a0f0d2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f9754b3c-7f81-4d15-8726-14659a0f0d2b",
      "name": "f9754b3c-7f81-4d15-8726-14659a0f0d2b",
      "status": "Failed",
      "startTime": "2026-10-02T14:15:50.839633613Z",
      "endTime": "2026-10-02T14:44:52.180223552Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc002486980>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4z7j926fljlc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f9754b3c-7f81-4d15-8726-14659a0f0d2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f9754b3c-7f81-4d15-8726-14659a0f0d2b",
      "name": "f9754b3c-7f81-4d15-8726-14659a0f0d2b",
      "status": "Failed",
      "startTime": "2026-10-02T14:15:50.839633613Z",
      "endTime": "2026-10-02T14:44:52.180223552Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-48fc7bfssxlp/cluster-candidate-4-22-7klbxb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d40740>: 
    failed to create HCP cluster cluster-candidate-4-22-7klbxb: failed waiting for cluster="cluster-candidate-4-22-7klbxb" in resourcegroup="rg-candidate-4-22-48fc7bfssxlp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3f2efc0e-134f-4975-8954-ac850a668709
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3f2efc0e-134f-4975-8954-ac850a668709",
      "name": "3f2efc0e-134f-4975-8954-ac850a668709",
      "status": "Failed",
      "startTime": "2026-10-02T14:48:41.860181794Z",
      "endTime": "2026-10-02T15:17:41.529062952Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-48fc7bfssxlp/cluster-candidate-4-22-7klbxb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d40740>: 
    failed to create HCP cluster cluster-candidate-4-22-7klbxb: failed waiting for cluster="cluster-candidate-4-22-7klbxb" in resourcegroup="rg-candidate-4-22-48fc7bfssxlp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3f2efc0e-134f-4975-8954-ac850a668709
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3f2efc0e-134f-4975-8954-ac850a668709",
      "name": "3f2efc0e-134f-4975-8954-ac850a668709",
      "status": "Failed",
      "startTime": "2026-10-02T14:48:41.860181794Z",
      "endTime": "2026-10-02T15:17:41.529062952Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0010ae5a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-k4q5zpfgpr4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3c73c7ae-1999-4cb4-b92d-cbffe4356bef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3c73c7ae-1999-4cb4-b92d-cbffe4356bef",
      "name": "3c73c7ae-1999-4cb4-b92d-cbffe4356bef",
      "status": "Failed",
      "startTime": "2026-10-02T14:14:49.716994175Z",
      "endTime": "2026-10-02T14:43:52.152352909Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0010ae5a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-k4q5zpfgpr4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3c73c7ae-1999-4cb4-b92d-cbffe4356bef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3c73c7ae-1999-4cb4-b92d-cbffe4356bef",
      "name": "3c73c7ae-1999-4cb4-b92d-cbffe4356bef",
      "status": "Failed",
      "startTime": "2026-10-02T14:14:49.716994175Z",
      "endTime": "2026-10-02T14:43:52.152352909Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-rf66b2pjnzs5/cluster-candidate-4-21-zp7tkm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a9f080>: 
    failed to create HCP cluster cluster-candidate-4-21-zp7tkm: failed waiting for cluster="cluster-candidate-4-21-zp7tkm" in resourcegroup="rg-candidate-4-21-rf66b2pjnzs5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f866fdfa-caa6-4c46-a375-3e94e0748ca1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f866fdfa-caa6-4c46-a375-3e94e0748ca1",
      "name": "f866fdfa-caa6-4c46-a375-3e94e0748ca1",
      "status": "Failed",
      "startTime": "2026-10-02T14:48:13.433342194Z",
      "endTime": "2026-10-02T15:17:21.452859562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-rf66b2pjnzs5/cluster-candidate-4-21-zp7tkm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a9f080>: 
    failed to create HCP cluster cluster-candidate-4-21-zp7tkm: failed waiting for cluster="cluster-candidate-4-21-zp7tkm" in resourcegroup="rg-candidate-4-21-rf66b2pjnzs5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f866fdfa-caa6-4c46-a375-3e94e0748ca1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f866fdfa-caa6-4c46-a375-3e94e0748ca1",
      "name": "f866fdfa-caa6-4c46-a375-3e94e0748ca1",
      "status": "Failed",
      "startTime": "2026-10-02T14:48:13.433342194Z",
      "endTime": "2026-10-02T15:17:21.452859562Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc002f205e0>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc002f205e0>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00105d400>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tqrndt5v6zjd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/8281d8b1-ac73-4bd5-b7ff-5e1ee0313132
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/8281d8b1-ac73-4bd5-b7ff-5e1ee0313132",
      "name": "8281d8b1-ac73-4bd5-b7ff-5e1ee0313132",
      "status": "Failed",
      "startTime": "2026-10-02T14:42:30.687332757Z",
      "endTime": "2026-10-02T15:01:32.464297541Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tqrndt5v6zjd--managed/providers/Microsoft.Compute/virtualMachines/x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tqrndt5v6zjd--managed/providers/Microsoft.Compute/virtualMachines/x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00105d400>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tqrndt5v6zjd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/8281d8b1-ac73-4bd5-b7ff-5e1ee0313132
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/8281d8b1-ac73-4bd5-b7ff-5e1ee0313132",
      "name": "8281d8b1-ac73-4bd5-b7ff-5e1ee0313132",
      "status": "Failed",
      "startTime": "2026-10-02T14:42:30.687332757Z",
      "endTime": "2026-10-02T15:01:32.464297541Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tqrndt5v6zjd--managed/providers/Microsoft.Compute/virtualMachines/x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tqrndt5v6zjd--managed/providers/Microsoft.Compute/virtualMachines/x0e7s2g7c7i0z9g-np-1-cpzvz-fbtjm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc0006eb260>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc0006eb260>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-fnp94g" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0017b8120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-pqm748zt4v25/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-fnp94g
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-fnp94g" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0017b8120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-pqm748zt4v25/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-fnp94g
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-phqn4q" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0014ae240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-krv2vhb295c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-phqn4q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-phqn4q" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0014ae240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-krv2vhb295c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-phqn4q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-zrd7b7" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0010807b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-pn6t6n4tjrff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-zrd7b7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-zrd7b7" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0010807b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-pn6t6n4tjrff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-zrd7b7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-9p29lj" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00153a060>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-l7kx94q698fp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9p29lj
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-9p29lj" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00153a060>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-l7kx94q698fp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9p29lj
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xmqw4g" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00155a450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-v87rt9mg748g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xmqw4g
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xmqw4g" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00155a450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-v87rt9mg748g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xmqw4g
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-27sjww at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000d81140>: 
    failed to create HCP cluster np-skip-minor-27sjww: failed waiting for cluster="np-skip-minor-27sjww" in resourcegroup="rg-np-skip-minor-27sjww-kgw6mbppvdmz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/74064ef6-783f-4e23-b96a-a2576b7c7d5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/74064ef6-783f-4e23-b96a-a2576b7c7d5f",
      "name": "74064ef6-783f-4e23-b96a-a2576b7c7d5f",
      "status": "Failed",
      "startTime": "2026-10-02T23:50:41.341932888Z",
      "endTime": "2026-10-03T00:19:42.833586106Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-27sjww at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000d81140>: 
    failed to create HCP cluster np-skip-minor-27sjww: failed waiting for cluster="np-skip-minor-27sjww" in resourcegroup="rg-np-skip-minor-27sjww-kgw6mbppvdmz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/74064ef6-783f-4e23-b96a-a2576b7c7d5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/74064ef6-783f-4e23-b96a-a2576b7c7d5f",
      "name": "74064ef6-783f-4e23-b96a-a2576b7c7d5f",
      "status": "Failed",
      "startTime": "2026-10-02T23:50:41.341932888Z",
      "endTime": "2026-10-03T00:19:42.833586106Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-8cm72gr5ql2s/cluster-nightly-4-22-65xxws should provision
Unexpected error:
    <*fmt.wrapError | 0xc00164c880>: 
    failed to create HCP cluster cluster-nightly-4-22-65xxws: failed waiting for cluster="cluster-nightly-4-22-65xxws" in resourcegroup="rg-nightly-4-22-8cm72gr5ql2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/562de0b5-c401-4c2c-b9be-9fd3eaaa9a92
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/562de0b5-c401-4c2c-b9be-9fd3eaaa9a92",
      "name": "562de0b5-c401-4c2c-b9be-9fd3eaaa9a92",
      "status": "Failed",
      "startTime": "2026-10-02T23:28:43.258428047Z",
      "endTime": "2026-10-02T23:57:43.738727793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-8cm72gr5ql2s/cluster-nightly-4-22-65xxws should provision
Unexpected error:
    <*fmt.wrapError | 0xc00164c880>: 
    failed to create HCP cluster cluster-nightly-4-22-65xxws: failed waiting for cluster="cluster-nightly-4-22-65xxws" in resourcegroup="rg-nightly-4-22-8cm72gr5ql2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/562de0b5-c401-4c2c-b9be-9fd3eaaa9a92
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/562de0b5-c401-4c2c-b9be-9fd3eaaa9a92",
      "name": "562de0b5-c401-4c2c-b9be-9fd3eaaa9a92",
      "status": "Failed",
      "startTime": "2026-10-02T23:28:43.258428047Z",
      "endTime": "2026-10-02T23:57:43.738727793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00106c720>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-88jnzt6h65n4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00106c720>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-88jnzt6h65n4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-766n78" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0xc00149e390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb9p7vsl4pjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-766n78
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-766n78" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0xc00149e390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb9p7vsl4pjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-766n78
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc00182a270>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7bxvbnqqpfz2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc00182a270>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7bxvbnqqpfz2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-p76rlg" to 4.21.0-0.nightly-multi-2026-10-02-021649
Unexpected error:
    <*exported.ResponseError | 0xc000ab88a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-c4g7bpjwkcrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-p76rlg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-p76rlg" to 4.21.0-0.nightly-multi-2026-10-02-021649
Unexpected error:
    <*exported.ResponseError | 0xc000ab88a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-c4g7bpjwkcrc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-p76rlg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tqrvbk" for cluster "cluster-nightly-5-0-tqrvbk"
Unexpected error:
    <*fmt.wrapError | 0xc000f91320>: 
    failed to create NodePool np-tqrvbk: failed waiting for nodepool="np-tqrvbk" for cluster "cluster-nightly-5-0-tqrvbk" in resourcegroup="rg-nightly-5-0-r6nmqb54mssw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/096e1e24-3a65-44de-9802-0a6339605f8f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/096e1e24-3a65-44de-9802-0a6339605f8f",
      "name": "096e1e24-3a65-44de-9802-0a6339605f8f",
      "status": "Failed",
      "startTime": "2026-10-02T23:41:44.73235146Z",
      "endTime": "2026-10-03T00:00:53.838500072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-89fp6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-d94nh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-89fp6: Creating: virtualmachine creating or updating\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-d94nh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tqrvbk" for cluster "cluster-nightly-5-0-tqrvbk"
Unexpected error:
    <*fmt.wrapError | 0xc000f91320>: 
    failed to create NodePool np-tqrvbk: failed waiting for nodepool="np-tqrvbk" for cluster "cluster-nightly-5-0-tqrvbk" in resourcegroup="rg-nightly-5-0-r6nmqb54mssw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/096e1e24-3a65-44de-9802-0a6339605f8f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/096e1e24-3a65-44de-9802-0a6339605f8f",
      "name": "096e1e24-3a65-44de-9802-0a6339605f8f",
      "status": "Failed",
      "startTime": "2026-10-02T23:41:44.73235146Z",
      "endTime": "2026-10-03T00:00:53.838500072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-89fp6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-d94nh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-89fp6: Creating: virtualmachine creating or updating\nMachine s0b5t8z5y1y3a5r-np-tqrvbk-zdnfc-d94nh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-lg5p6f" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a56dc0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-lg5p6f: failed waiting for cluster="cp-ystream-upgrade-5-0-lg5p6f" in resourcegroup="rg-cp-ystream-upgrade-5-0-b2ldx28lrgzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c7f2efd-8245-4407-9270-f3937d54358b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c7f2efd-8245-4407-9270-f3937d54358b",
      "name": "9c7f2efd-8245-4407-9270-f3937d54358b",
      "status": "Failed",
      "startTime": "2026-10-02T23:32:39.04806912Z",
      "endTime": "2026-10-03T00:01:43.756113041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-lg5p6f" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a56dc0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-lg5p6f: failed waiting for cluster="cp-ystream-upgrade-5-0-lg5p6f" in resourcegroup="rg-cp-ystream-upgrade-5-0-b2ldx28lrgzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c7f2efd-8245-4407-9270-f3937d54358b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c7f2efd-8245-4407-9270-f3937d54358b",
      "name": "9c7f2efd-8245-4407-9270-f3937d54358b",
      "status": "Failed",
      "startTime": "2026-10-02T23:32:39.04806912Z",
      "endTime": "2026-10-03T00:01:43.756113041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc00141a900>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-kmfrpjr862hw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kmfrpjr862hw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xc00141a900>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-kmfrpjr862hw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kmfrpjr862hw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010647a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-b4pbcj49h2gk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56464cc9-6bea-4036-8793-830b73f9c445
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56464cc9-6bea-4036-8793-830b73f9c445",
      "name": "56464cc9-6bea-4036-8793-830b73f9c445",
      "status": "Failed",
      "startTime": "2026-10-02T23:18:22.317106803Z",
      "endTime": "2026-10-02T23:37:23.768654646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-882wt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-t6wms: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-882wt: Creating: virtualmachine creating or updating\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-t6wms: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010647a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-b4pbcj49h2gk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56464cc9-6bea-4036-8793-830b73f9c445
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56464cc9-6bea-4036-8793-830b73f9c445",
      "name": "56464cc9-6bea-4036-8793-830b73f9c445",
      "status": "Failed",
      "startTime": "2026-10-02T23:18:22.317106803Z",
      "endTime": "2026-10-02T23:37:23.768654646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-882wt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-t6wms: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-882wt: Creating: virtualmachine creating or updating\nMachine z5o3s0b7q4a6b4y-np-1-zmwpp-t6wms: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-85ljr8" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc001476720>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-85ljr8: failed waiting for cluster="cp-ystream-upgrade-4-23-85ljr8" in resourcegroup="rg-cp-ystream-upgrade-4-23-xxm22554wqqw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4209a33e-d8fd-4e71-a9e2-6938fd555838
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4209a33e-d8fd-4e71-a9e2-6938fd555838",
      "name": "4209a33e-d8fd-4e71-a9e2-6938fd555838",
      "status": "Failed",
      "startTime": "2026-10-02T23:32:20.614033798Z",
      "endTime": "2026-10-03T00:01:23.755103284Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-85ljr8" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc001476720>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-85ljr8: failed waiting for cluster="cp-ystream-upgrade-4-23-85ljr8" in resourcegroup="rg-cp-ystream-upgrade-4-23-xxm22554wqqw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4209a33e-d8fd-4e71-a9e2-6938fd555838
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4209a33e-d8fd-4e71-a9e2-6938fd555838",
      "name": "4209a33e-d8fd-4e71-a9e2-6938fd555838",
      "status": "Failed",
      "startTime": "2026-10-02T23:32:20.614033798Z",
      "endTime": "2026-10-03T00:01:23.755103284Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00100c018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-dbjg2z24dvcx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2f88e2f7-11dc-4604-9894-6c4ed35e59a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2f88e2f7-11dc-4604-9894-6c4ed35e59a2",
      "name": "2f88e2f7-11dc-4604-9894-6c4ed35e59a2",
      "status": "Failed",
      "startTime": "2026-10-02T23:17:46.069336916Z",
      "endTime": "2026-10-02T23:36:53.678900847Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-p829p-2b2s6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-p829p-t26hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-p829p-2b2s6: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-p829p-t26hm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00100c018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-dbjg2z24dvcx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2f88e2f7-11dc-4604-9894-6c4ed35e59a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2f88e2f7-11dc-4604-9894-6c4ed35e59a2",
      "name": "2f88e2f7-11dc-4604-9894-6c4ed35e59a2",
      "status": "Failed",
      "startTime": "2026-10-02T23:17:46.069336916Z",
      "endTime": "2026-10-02T23:36:53.678900847Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-p829p-2b2s6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-p829p-t26hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-p829p-2b2s6: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-p829p-t26hm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fpzhbk" for cluster "cluster-nightly-4-21-fpzhbk"
Unexpected error:
    <*fmt.wrapError | 0xc000c80140>: 
    failed to create NodePool np-fpzhbk: failed waiting for nodepool="np-fpzhbk" for cluster "cluster-nightly-4-21-fpzhbk" in resourcegroup="rg-nightly-4-21-lf2wqrcgx8mk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f07e721e-1ddd-4cdc-848f-f939d97b8b6a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f07e721e-1ddd-4cdc-848f-f939d97b8b6a",
      "name": "f07e721e-1ddd-4cdc-848f-f939d97b8b6a",
      "status": "Failed",
      "startTime": "2026-10-02T23:36:50.854597667Z",
      "endTime": "2026-10-02T23:55:53.768834785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-g4dx8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-z5xht: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-g4dx8: Creating: virtualmachine creating or updating\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-z5xht: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fpzhbk" for cluster "cluster-nightly-4-21-fpzhbk"
Unexpected error:
    <*fmt.wrapError | 0xc000c80140>: 
    failed to create NodePool np-fpzhbk: failed waiting for nodepool="np-fpzhbk" for cluster "cluster-nightly-4-21-fpzhbk" in resourcegroup="rg-nightly-4-21-lf2wqrcgx8mk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f07e721e-1ddd-4cdc-848f-f939d97b8b6a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f07e721e-1ddd-4cdc-848f-f939d97b8b6a",
      "name": "f07e721e-1ddd-4cdc-848f-f939d97b8b6a",
      "status": "Failed",
      "startTime": "2026-10-02T23:36:50.854597667Z",
      "endTime": "2026-10-02T23:55:53.768834785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-g4dx8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-z5xht: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-g4dx8: Creating: virtualmachine creating or updating\nMachine v2x8c0w5w4x9u4j-np-fpzhbk-pj8tx-z5xht: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-gfwqg6" to 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*exported.ResponseError | 0xc0018044b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-p7jzbr5ffl22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gfwqg6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-gfwqg6" to 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*exported.ResponseError | 0xc0018044b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-p7jzbr5ffl22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gfwqg6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc001214a20>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-r44ftv" in resourcegroup="rg-np-version-upgrade-r44ftv-dq6wfvlkhtqj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r44ftv-dq6wfvlkhtqj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-r44ftv/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc001214a20>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-r44ftv" in resourcegroup="rg-np-version-upgrade-r44ftv-dq6wfvlkhtqj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r44ftv-dq6wfvlkhtqj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-r44ftv/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0005e2c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-9rcfwwnss4kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0005e2c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-9rcfwwnss4kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc002910ff0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-rbgmrr9dbzzr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc002910ff0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-rbgmrr9dbzzr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc002b293e0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rx57wpbdxhdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/811670f5-5dbe-439c-9697-edae7243b031
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/811670f5-5dbe-439c-9697-edae7243b031",
      "name": "811670f5-5dbe-439c-9697-edae7243b031",
      "status": "Failed",
      "startTime": "2026-10-02T23:47:39.993818098Z",
      "endTime": "2026-10-03T00:06:42.823230365Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-lgvpd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-thccv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-lgvpd: Creating: virtualmachine creating or updating\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-thccv: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc002b293e0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rx57wpbdxhdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/811670f5-5dbe-439c-9697-edae7243b031
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/811670f5-5dbe-439c-9697-edae7243b031",
      "name": "811670f5-5dbe-439c-9697-edae7243b031",
      "status": "Failed",
      "startTime": "2026-10-02T23:47:39.993818098Z",
      "endTime": "2026-10-03T00:06:42.823230365Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-lgvpd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-thccv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-lgvpd: Creating: virtualmachine creating or updating\nMachine e9p1n9f9j7n4p2s-np-delete-main-ntkpd-thccv: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000aae3e0>: 
    failed to create HCP cluster np-dg-minor-t84576: failed waiting for cluster="np-dg-minor-t84576" in resourcegroup="rg-np-dg-minor-t84576-xhl45nnpkcxs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86a92082-52bb-4ade-8c3c-f2f5320c6001
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86a92082-52bb-4ade-8c3c-f2f5320c6001",
      "name": "86a92082-52bb-4ade-8c3c-f2f5320c6001",
      "status": "Failed",
      "startTime": "2026-10-02T23:50:46.033712456Z",
      "endTime": "2026-10-03T00:19:52.83457796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000aae3e0>: 
    failed to create HCP cluster np-dg-minor-t84576: failed waiting for cluster="np-dg-minor-t84576" in resourcegroup="rg-np-dg-minor-t84576-xhl45nnpkcxs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86a92082-52bb-4ade-8c3c-f2f5320c6001
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86a92082-52bb-4ade-8c3c-f2f5320c6001",
      "name": "86a92082-52bb-4ade-8c3c-f2f5320c6001",
      "status": "Failed",
      "startTime": "2026-10-02T23:50:46.033712456Z",
      "endTime": "2026-10-03T00:19:52.83457796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc00089caa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-b7c2z62bpd6r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c057ce9-73bf-4556-855a-010dcaa301e1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c057ce9-73bf-4556-855a-010dcaa301e1",
      "name": "0c057ce9-73bf-4556-855a-010dcaa301e1",
      "status": "Failed",
      "startTime": "2026-10-02T23:15:33.73277988Z",
      "endTime": "2026-10-02T23:34:33.89328902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-lcxp7-kcfwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-b7c2z62bpd6r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-lcxp7-kcfwr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-lcxp7-kcfwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-b7c2z62bpd6r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-lcxp7-kcfwr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc00089caa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-b7c2z62bpd6r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c057ce9-73bf-4556-855a-010dcaa301e1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c057ce9-73bf-4556-855a-010dcaa301e1",
      "name": "0c057ce9-73bf-4556-855a-010dcaa301e1",
      "status": "Failed",
      "startTime": "2026-10-02T23:15:33.73277988Z",
      "endTime": "2026-10-02T23:34:33.89328902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-lcxp7-kcfwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-b7c2z62bpd6r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-lcxp7-kcfwr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-lcxp7-kcfwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-b7c2z62bpd6r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-lcxp7-kcfwr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vw9ss6" for cluster "cluster-nightly-4-23-vw9ss6"
Unexpected error:
    <*fmt.wrapError | 0xc000e0f2c0>: 
    failed to create NodePool np-vw9ss6: failed waiting for nodepool="np-vw9ss6" for cluster "cluster-nightly-4-23-vw9ss6" in resourcegroup="rg-nightly-4-23-66zfk978pxch" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04cd216e-5e0e-4e76-a1ab-6ea26fbb70a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04cd216e-5e0e-4e76-a1ab-6ea26fbb70a0",
      "name": "04cd216e-5e0e-4e76-a1ab-6ea26fbb70a0",
      "status": "Failed",
      "startTime": "2026-10-02T23:42:21.716380971Z",
      "endTime": "2026-10-03T00:01:23.832580289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-qhsr8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-wcqs9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-qhsr8: Creating: virtualmachine creating or updating\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-wcqs9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vw9ss6" for cluster "cluster-nightly-4-23-vw9ss6"
Unexpected error:
    <*fmt.wrapError | 0xc000e0f2c0>: 
    failed to create NodePool np-vw9ss6: failed waiting for nodepool="np-vw9ss6" for cluster "cluster-nightly-4-23-vw9ss6" in resourcegroup="rg-nightly-4-23-66zfk978pxch" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04cd216e-5e0e-4e76-a1ab-6ea26fbb70a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04cd216e-5e0e-4e76-a1ab-6ea26fbb70a0",
      "name": "04cd216e-5e0e-4e76-a1ab-6ea26fbb70a0",
      "status": "Failed",
      "startTime": "2026-10-02T23:42:21.716380971Z",
      "endTime": "2026-10-03T00:01:23.832580289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-qhsr8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-wcqs9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-qhsr8: Creating: virtualmachine creating or updating\nMachine h7d3c7e5o0i6e7s-np-vw9ss6-lw4g2-wcqs9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rx699z" for cluster "cluster-nightly-5-1-rx699z"
Unexpected error:
    <*fmt.wrapError | 0xc000eb8200>: 
    failed to create NodePool np-rx699z: failed waiting for nodepool="np-rx699z" for cluster "cluster-nightly-5-1-rx699z" in resourcegroup="rg-nightly-5-1-x6gpqlf2wmlk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ac35c1-1131-4bbe-9ed8-04f87748743e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ac35c1-1131-4bbe-9ed8-04f87748743e",
      "name": "13ac35c1-1131-4bbe-9ed8-04f87748743e",
      "status": "Failed",
      "startTime": "2026-10-02T23:40:19.915190927Z",
      "endTime": "2026-10-02T23:59:23.95618655Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-d28hs: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-kswmm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-d28hs: Creating: virtualmachine creating or updating\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-kswmm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rx699z" for cluster "cluster-nightly-5-1-rx699z"
Unexpected error:
    <*fmt.wrapError | 0xc000eb8200>: 
    failed to create NodePool np-rx699z: failed waiting for nodepool="np-rx699z" for cluster "cluster-nightly-5-1-rx699z" in resourcegroup="rg-nightly-5-1-x6gpqlf2wmlk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ac35c1-1131-4bbe-9ed8-04f87748743e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ac35c1-1131-4bbe-9ed8-04f87748743e",
      "name": "13ac35c1-1131-4bbe-9ed8-04f87748743e",
      "status": "Failed",
      "startTime": "2026-10-02T23:40:19.915190927Z",
      "endTime": "2026-10-02T23:59:23.95618655Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-d28hs: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-kswmm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-d28hs: Creating: virtualmachine creating or updating\nMachine t2t5i8t7n0a2i6q-np-rx699z-9wkrd-kswmm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001680330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-kcc2h87qnl8v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001680330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-kcc2h87qnl8v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-f6fpxn" for cluster "cluster-nightly-4-20-f6fpxn"
Unexpected error:
    <*fmt.wrapError | 0xc000be25e0>: 
    failed to create NodePool np-f6fpxn: failed starting nodepool creation "np-f6fpxn" for cluster "cluster-nightly-4-20-f6fpxn" in resourcegroup="rg-nightly-4-20-9z2m2dsr8b8d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-9z2m2dsr8b8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-f6fpxn/nodePools/np-f6fpxn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-f6fpxn" for cluster "cluster-nightly-4-20-f6fpxn"
Unexpected error:
    <*fmt.wrapError | 0xc000be25e0>: 
    failed to create NodePool np-f6fpxn: failed starting nodepool creation "np-f6fpxn" for cluster "cluster-nightly-4-20-f6fpxn" in resourcegroup="rg-nightly-4-20-9z2m2dsr8b8d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-9z2m2dsr8b8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-f6fpxn/nodePools/np-f6fpxn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0xc0010a36c0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-tsckb2km58j7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-tsckb2km58j7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0xc0010a36c0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-tsckb2km58j7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-tsckb2km58j7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k6xcnn" for cluster "cluster-candidate-5-1-k6xcnn"
Unexpected error:
    <*fmt.wrapError | 0xc000a95460>: 
    failed to create NodePool np-k6xcnn: failed waiting for nodepool="np-k6xcnn" for cluster "cluster-candidate-5-1-k6xcnn" in resourcegroup="rg-candidate-5-1-xcbcjc2xvxnz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4a6f0d3-f2f8-4021-bdab-85f57fe8a1f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4a6f0d3-f2f8-4021-bdab-85f57fe8a1f8",
      "name": "f4a6f0d3-f2f8-4021-bdab-85f57fe8a1f8",
      "status": "Failed",
      "startTime": "2026-10-02T21:05:55.776603892Z",
      "endTime": "2026-10-02T21:24:58.213196373Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k6xcnn" for cluster "cluster-candidate-5-1-k6xcnn"
Unexpected error:
    <*fmt.wrapError | 0xc000a95460>: 
    failed to create NodePool np-k6xcnn: failed waiting for nodepool="np-k6xcnn" for cluster "cluster-candidate-5-1-k6xcnn" in resourcegroup="rg-candidate-5-1-xcbcjc2xvxnz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4a6f0d3-f2f8-4021-bdab-85f57fe8a1f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4a6f0d3-f2f8-4021-bdab-85f57fe8a1f8",
      "name": "f4a6f0d3-f2f8-4021-bdab-85f57fe8a1f8",
      "status": "Failed",
      "startTime": "2026-10-02T21:05:55.776603892Z",
      "endTime": "2026-10-02T21:24:58.213196373Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001880060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-flb2g5s6xxjr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3afe0f8-5077-4a5f-b1fe-943343d11f7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3afe0f8-5077-4a5f-b1fe-943343d11f7a",
      "name": "c3afe0f8-5077-4a5f-b1fe-943343d11f7a",
      "status": "Failed",
      "startTime": "2026-10-02T20:40:40.455432211Z",
      "endTime": "2026-10-02T20:59:49.144387256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-4kgx8-km75d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-4kgx8-wjql6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-4kgx8-km75d: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-4kgx8-wjql6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001880060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-flb2g5s6xxjr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3afe0f8-5077-4a5f-b1fe-943343d11f7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3afe0f8-5077-4a5f-b1fe-943343d11f7a",
      "name": "c3afe0f8-5077-4a5f-b1fe-943343d11f7a",
      "status": "Failed",
      "startTime": "2026-10-02T20:40:40.455432211Z",
      "endTime": "2026-10-02T20:59:49.144387256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-4kgx8-km75d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-4kgx8-wjql6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-4kgx8-km75d: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-4kgx8-wjql6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rfrkgm" for cluster "cluster-candidate-4-22-rfrkgm"
Unexpected error:
    <*fmt.wrapError | 0xc000a0efa0>: 
    failed to create NodePool np-rfrkgm: failed waiting for nodepool="np-rfrkgm" for cluster "cluster-candidate-4-22-rfrkgm" in resourcegroup="rg-candidate-4-22-4dnctl8qsdmb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd66a171-5708-426c-a085-dc7fbb337a5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd66a171-5708-426c-a085-dc7fbb337a5f",
      "name": "bd66a171-5708-426c-a085-dc7fbb337a5f",
      "status": "Failed",
      "startTime": "2026-10-02T21:02:30.361632741Z",
      "endTime": "2026-10-02T21:21:38.229702877Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-dgcdt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-z5pzp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-dgcdt: Creating: virtualmachine creating or updating\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-z5pzp: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rfrkgm" for cluster "cluster-candidate-4-22-rfrkgm"
Unexpected error:
    <*fmt.wrapError | 0xc000a0efa0>: 
    failed to create NodePool np-rfrkgm: failed waiting for nodepool="np-rfrkgm" for cluster "cluster-candidate-4-22-rfrkgm" in resourcegroup="rg-candidate-4-22-4dnctl8qsdmb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd66a171-5708-426c-a085-dc7fbb337a5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd66a171-5708-426c-a085-dc7fbb337a5f",
      "name": "bd66a171-5708-426c-a085-dc7fbb337a5f",
      "status": "Failed",
      "startTime": "2026-10-02T21:02:30.361632741Z",
      "endTime": "2026-10-02T21:21:38.229702877Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-dgcdt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-z5pzp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-dgcdt: Creating: virtualmachine creating or updating\nMachine z0t1i4f5k3c2g3w-np-rfrkgm-drz4w-z5pzp: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fbb780>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-9v5cc2pldl4b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/64c53475-70e9-4ead-ac9d-b255c14e111e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/64c53475-70e9-4ead-ac9d-b255c14e111e",
      "name": "64c53475-70e9-4ead-ac9d-b255c14e111e",
      "status": "Failed",
      "startTime": "2026-10-02T20:37:10.986126021Z",
      "endTime": "2026-10-02T20:56:19.371811745Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-9v5cc2pldl4b--managed/providers/Microsoft.Compute/virtualMachines/v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-9v5cc2pldl4b--managed/providers/Microsoft.Compute/virtualMachines/v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fbb780>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-9v5cc2pldl4b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/64c53475-70e9-4ead-ac9d-b255c14e111e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/64c53475-70e9-4ead-ac9d-b255c14e111e",
      "name": "64c53475-70e9-4ead-ac9d-b255c14e111e",
      "status": "Failed",
      "startTime": "2026-10-02T20:37:10.986126021Z",
      "endTime": "2026-10-02T20:56:19.371811745Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-9v5cc2pldl4b--managed/providers/Microsoft.Compute/virtualMachines/v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-9v5cc2pldl4b--managed/providers/Microsoft.Compute/virtualMachines/v2z0r1j6g1i1v5p-np-1-pkgzx-qrzhh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c2b720>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-fgzfw6brhrkb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6270f4b8-a6c4-4356-a32a-feb00b0c9403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6270f4b8-a6c4-4356-a32a-feb00b0c9403",
      "name": "6270f4b8-a6c4-4356-a32a-feb00b0c9403",
      "status": "Failed",
      "startTime": "2026-10-02T21:34:15.426377086Z",
      "endTime": "2026-10-02T21:53:18.348312497Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-8sclt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-ksnwd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-8sclt: Creating: virtualmachine creating or updating\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-ksnwd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c2b720>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-fgzfw6brhrkb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6270f4b8-a6c4-4356-a32a-feb00b0c9403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6270f4b8-a6c4-4356-a32a-feb00b0c9403",
      "name": "6270f4b8-a6c4-4356-a32a-feb00b0c9403",
      "status": "Failed",
      "startTime": "2026-10-02T21:34:15.426377086Z",
      "endTime": "2026-10-02T21:53:18.348312497Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-8sclt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-ksnwd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-8sclt: Creating: virtualmachine creating or updating\nMachine p8k4o5c9o8v2d0a-np-deletable-9k5ld-ksnwd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000f04d60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-j6tfrrstctwj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2237f1ea-36ac-4925-ab4c-3bf34b150641
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2237f1ea-36ac-4925-ab4c-3bf34b150641",
      "name": "2237f1ea-36ac-4925-ab4c-3bf34b150641",
      "status": "Failed",
      "startTime": "2026-10-02T20:40:29.897644584Z",
      "endTime": "2026-10-02T20:59:29.402047255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-krs6w-gdbz8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-krs6w-gdbz8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000f04d60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-j6tfrrstctwj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2237f1ea-36ac-4925-ab4c-3bf34b150641
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2237f1ea-36ac-4925-ab4c-3bf34b150641",
      "name": "2237f1ea-36ac-4925-ab4c-3bf34b150641",
      "status": "Failed",
      "startTime": "2026-10-02T20:40:29.897644584Z",
      "endTime": "2026-10-02T20:59:29.402047255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-krs6w-gdbz8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-krs6w-gdbz8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qpmglj" for cluster "cluster-candidate-4-20-qpmglj"
Unexpected error:
    <*fmt.wrapError | 0xc00128af20>: 
    failed to create NodePool np-qpmglj: failed waiting for nodepool="np-qpmglj" for cluster "cluster-candidate-4-20-qpmglj" in resourcegroup="rg-candidate-4-20-lrjrfzqq46lq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f59c352-b15a-44fe-8eb1-052155b6abc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f59c352-b15a-44fe-8eb1-052155b6abc3",
      "name": "8f59c352-b15a-44fe-8eb1-052155b6abc3",
      "status": "Failed",
      "startTime": "2026-10-02T21:01:08.060624103Z",
      "endTime": "2026-10-02T21:20:08.251497411Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-pngl6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-rh562: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-pngl6: Creating: virtualmachine creating or updating\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-rh562: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qpmglj" for cluster "cluster-candidate-4-20-qpmglj"
Unexpected error:
    <*fmt.wrapError | 0xc00128af20>: 
    failed to create NodePool np-qpmglj: failed waiting for nodepool="np-qpmglj" for cluster "cluster-candidate-4-20-qpmglj" in resourcegroup="rg-candidate-4-20-lrjrfzqq46lq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f59c352-b15a-44fe-8eb1-052155b6abc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f59c352-b15a-44fe-8eb1-052155b6abc3",
      "name": "8f59c352-b15a-44fe-8eb1-052155b6abc3",
      "status": "Failed",
      "startTime": "2026-10-02T21:01:08.060624103Z",
      "endTime": "2026-10-02T21:20:08.251497411Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-pngl6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-rh562: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-pngl6: Creating: virtualmachine creating or updating\nMachine a7s6c5x0k3i6u9a-np-qpmglj-tcqmc-rh562: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001172048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-5sg2h42pm8fv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ac0d101-3fd0-42ea-ae32-a6afba54d453
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ac0d101-3fd0-42ea-ae32-a6afba54d453",
      "name": "4ac0d101-3fd0-42ea-ae32-a6afba54d453",
      "status": "Failed",
      "startTime": "2026-10-02T20:43:49.738242165Z",
      "endTime": "2026-10-02T21:02:49.398233298Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-wg9bd-twwb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-wg9bd-z9q58: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-wg9bd-twwb6: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-wg9bd-z9q58: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001172048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-5sg2h42pm8fv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ac0d101-3fd0-42ea-ae32-a6afba54d453
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4ac0d101-3fd0-42ea-ae32-a6afba54d453",
      "name": "4ac0d101-3fd0-42ea-ae32-a6afba54d453",
      "status": "Failed",
      "startTime": "2026-10-02T20:43:49.738242165Z",
      "endTime": "2026-10-02T21:02:49.398233298Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-wg9bd-twwb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-wg9bd-z9q58: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-wg9bd-twwb6: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-wg9bd-z9q58: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f6f0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-cccnsncfs2r4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c87a1bea-d45c-40f0-b36e-76c1418b3181
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c87a1bea-d45c-40f0-b36e-76c1418b3181",
      "name": "c87a1bea-d45c-40f0-b36e-76c1418b3181",
      "status": "Failed",
      "startTime": "2026-10-02T20:39:19.772177296Z",
      "endTime": "2026-10-02T20:58:19.197731682Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-mcr5z: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-pn7ft: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-mcr5z: Creating: virtualmachine creating or updating\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-pn7ft: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f6f0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-cccnsncfs2r4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c87a1bea-d45c-40f0-b36e-76c1418b3181
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c87a1bea-d45c-40f0-b36e-76c1418b3181",
      "name": "c87a1bea-d45c-40f0-b36e-76c1418b3181",
      "status": "Failed",
      "startTime": "2026-10-02T20:39:19.772177296Z",
      "endTime": "2026-10-02T20:58:19.197731682Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-mcr5z: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-pn7ft: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-mcr5z: Creating: virtualmachine creating or updating\nMachine m9d7f6f9z0s9j7k-np-1-m4ndt-pn7ft: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-4rsvdk" for cluster "cluster-candidate-5-0-4rsvdk"
Unexpected error:
    <*fmt.wrapError | 0xc000ced660>: 
    failed to create NodePool np-4rsvdk: failed waiting for nodepool="np-4rsvdk" for cluster "cluster-candidate-5-0-4rsvdk" in resourcegroup="rg-candidate-5-0-mr9dnz4m68mv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe99a24d-45b0-4c02-b399-8a58c33f760f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe99a24d-45b0-4c02-b399-8a58c33f760f",
      "name": "fe99a24d-45b0-4c02-b399-8a58c33f760f",
      "status": "Failed",
      "startTime": "2026-10-02T21:03:59.509943483Z",
      "endTime": "2026-10-02T21:23:08.337275316Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gjgrx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gkprg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gjgrx: Creating: virtualmachine creating or updating\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gkprg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-4rsvdk" for cluster "cluster-candidate-5-0-4rsvdk"
Unexpected error:
    <*fmt.wrapError | 0xc000ced660>: 
    failed to create NodePool np-4rsvdk: failed waiting for nodepool="np-4rsvdk" for cluster "cluster-candidate-5-0-4rsvdk" in resourcegroup="rg-candidate-5-0-mr9dnz4m68mv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe99a24d-45b0-4c02-b399-8a58c33f760f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe99a24d-45b0-4c02-b399-8a58c33f760f",
      "name": "fe99a24d-45b0-4c02-b399-8a58c33f760f",
      "status": "Failed",
      "startTime": "2026-10-02T21:03:59.509943483Z",
      "endTime": "2026-10-02T21:23:08.337275316Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gjgrx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gkprg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gjgrx: Creating: virtualmachine creating or updating\nMachine g6y4a6l5u7h6m2s-np-4rsvdk-jgm75-gkprg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7lqqwg" for cluster "cluster-candidate-4-21-7lqqwg"
Unexpected error:
    <*fmt.wrapError | 0xc000d194a0>: 
    failed to create NodePool np-7lqqwg: failed waiting for nodepool="np-7lqqwg" for cluster "cluster-candidate-4-21-7lqqwg" in resourcegroup="rg-candidate-4-21-fsj5jdg9tljp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d395f2bf-88ca-4f84-8998-c863978caf3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d395f2bf-88ca-4f84-8998-c863978caf3f",
      "name": "d395f2bf-88ca-4f84-8998-c863978caf3f",
      "status": "Failed",
      "startTime": "2026-10-02T21:03:18.868079791Z",
      "endTime": "2026-10-02T21:22:18.599909683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7lqqwg" for cluster "cluster-candidate-4-21-7lqqwg"
Unexpected error:
    <*fmt.wrapError | 0xc000d194a0>: 
    failed to create NodePool np-7lqqwg: failed waiting for nodepool="np-7lqqwg" for cluster "cluster-candidate-4-21-7lqqwg" in resourcegroup="rg-candidate-4-21-fsj5jdg9tljp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d395f2bf-88ca-4f84-8998-c863978caf3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d395f2bf-88ca-4f84-8998-c863978caf3f",
      "name": "d395f2bf-88ca-4f84-8998-c863978caf3f",
      "status": "Failed",
      "startTime": "2026-10-02T21:03:18.868079791Z",
      "endTime": "2026-10-02T21:22:18.599909683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-878fr2" for cluster "cluster-candidate-4-23-878fr2"
Unexpected error:
    <*fmt.wrapError | 0xc000bb09a0>: 
    failed to create NodePool np-878fr2: failed waiting for nodepool="np-878fr2" for cluster "cluster-candidate-4-23-878fr2" in resourcegroup="rg-candidate-4-23-tq4tlj8sn8rt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d39da2e3-e8ed-4ecb-a802-520caf1d680a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d39da2e3-e8ed-4ecb-a802-520caf1d680a",
      "name": "d39da2e3-e8ed-4ecb-a802-520caf1d680a",
      "status": "Failed",
      "startTime": "2026-10-02T21:05:40.949801842Z",
      "endTime": "2026-10-02T21:24:48.213673897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-7czvr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-wcjsm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-7czvr: Creating: virtualmachine creating or updating\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-wcjsm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-878fr2" for cluster "cluster-candidate-4-23-878fr2"
Unexpected error:
    <*fmt.wrapError | 0xc000bb09a0>: 
    failed to create NodePool np-878fr2: failed waiting for nodepool="np-878fr2" for cluster "cluster-candidate-4-23-878fr2" in resourcegroup="rg-candidate-4-23-tq4tlj8sn8rt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d39da2e3-e8ed-4ecb-a802-520caf1d680a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d39da2e3-e8ed-4ecb-a802-520caf1d680a",
      "name": "d39da2e3-e8ed-4ecb-a802-520caf1d680a",
      "status": "Failed",
      "startTime": "2026-10-02T21:05:40.949801842Z",
      "endTime": "2026-10-02T21:24:48.213673897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-7czvr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-wcjsm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-7czvr: Creating: virtualmachine creating or updating\nMachine g4g6i1d1r4t2m8l-np-878fr2-58lpw-wcjsm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ee2740>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-ht5gkd" in resourcegroup="rg-np-noedge-ht5gkd-s9tjwqwzr9v2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2685dbe2-74b7-44fa-8866-0542656a2cbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2685dbe2-74b7-44fa-8866-0542656a2cbe",
      "name": "2685dbe2-74b7-44fa-8866-0542656a2cbe",
      "status": "Failed",
      "startTime": "2026-10-02T19:26:52.969260057Z",
      "endTime": "2026-10-02T19:45:54.294162813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ht5gkd-s9tjwqwzr9v2-np-ne-ht5gkd--managed/providers/Microsoft.Compute/virtualMachines/s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ht5gkd-s9tjwqwzr9v2-np-ne-ht5gkd--managed/providers/Microsoft.Compute/virtualMachines/s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ee2740>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-ht5gkd" in resourcegroup="rg-np-noedge-ht5gkd-s9tjwqwzr9v2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2685dbe2-74b7-44fa-8866-0542656a2cbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2685dbe2-74b7-44fa-8866-0542656a2cbe",
      "name": "2685dbe2-74b7-44fa-8866-0542656a2cbe",
      "status": "Failed",
      "startTime": "2026-10-02T19:26:52.969260057Z",
      "endTime": "2026-10-02T19:45:54.294162813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ht5gkd-s9tjwqwzr9v2-np-ne-ht5gkd--managed/providers/Microsoft.Compute/virtualMachines/s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ht5gkd-s9tjwqwzr9v2-np-ne-ht5gkd--managed/providers/Microsoft.Compute/virtualMachines/s8l8q3n7s8a7r2q-npnoedge-4-20-gplmr-p7qxl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-97jnb4" for cluster "cluster-candidate-4-22-97jnb4"
Unexpected error:
    <*fmt.wrapError | 0xc000a61960>: 
    failed to create NodePool np-97jnb4: failed waiting for nodepool="np-97jnb4" for cluster "cluster-candidate-4-22-97jnb4" in resourcegroup="rg-candidate-4-22-gqvl8n7h4bmq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d14fd998-aa3e-49c8-a405-91cc8c02a377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d14fd998-aa3e-49c8-a405-91cc8c02a377",
      "name": "d14fd998-aa3e-49c8-a405-91cc8c02a377",
      "status": "Failed",
      "startTime": "2026-10-02T18:49:32.578309403Z",
      "endTime": "2026-10-02T19:08:33.241090855Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-dkh4p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-hz2bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-dkh4p: Creating: virtualmachine creating or updating\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-hz2bh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-97jnb4" for cluster "cluster-candidate-4-22-97jnb4"
Unexpected error:
    <*fmt.wrapError | 0xc000a61960>: 
    failed to create NodePool np-97jnb4: failed waiting for nodepool="np-97jnb4" for cluster "cluster-candidate-4-22-97jnb4" in resourcegroup="rg-candidate-4-22-gqvl8n7h4bmq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d14fd998-aa3e-49c8-a405-91cc8c02a377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d14fd998-aa3e-49c8-a405-91cc8c02a377",
      "name": "d14fd998-aa3e-49c8-a405-91cc8c02a377",
      "status": "Failed",
      "startTime": "2026-10-02T18:49:32.578309403Z",
      "endTime": "2026-10-02T19:08:33.241090855Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-dkh4p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-hz2bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-dkh4p: Creating: virtualmachine creating or updating\nMachine v5r4w0l0u4k4n4f-np-97jnb4-k5gv8-hz2bh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001522060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nl8vdbhxwtvx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31e50407-d006-4e42-9498-e3bbc2ee0de1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31e50407-d006-4e42-9498-e3bbc2ee0de1",
      "name": "31e50407-d006-4e42-9498-e3bbc2ee0de1",
      "status": "Failed",
      "startTime": "2026-10-02T18:27:19.35370587Z",
      "endTime": "2026-10-02T18:46:22.521838138Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-h6sjn-cnj2s: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-h6sjn-h8wr9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-h6sjn-cnj2s: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-h6sjn-h8wr9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001522060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nl8vdbhxwtvx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31e50407-d006-4e42-9498-e3bbc2ee0de1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31e50407-d006-4e42-9498-e3bbc2ee0de1",
      "name": "31e50407-d006-4e42-9498-e3bbc2ee0de1",
      "status": "Failed",
      "startTime": "2026-10-02T18:27:19.35370587Z",
      "endTime": "2026-10-02T18:46:22.521838138Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-h6sjn-cnj2s: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-h6sjn-h8wr9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-h6sjn-cnj2s: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-h6sjn-h8wr9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kfvn25" for cluster "cluster-candidate-4-21-kfvn25"
Unexpected error:
    <*fmt.wrapError | 0xc000b3b220>: 
    failed to create NodePool np-kfvn25: failed waiting for nodepool="np-kfvn25" for cluster "cluster-candidate-4-21-kfvn25" in resourcegroup="rg-candidate-4-21-85wnrgqqbd2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b12e98e2-7f93-4004-87fa-b863b0f0044f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b12e98e2-7f93-4004-87fa-b863b0f0044f",
      "name": "b12e98e2-7f93-4004-87fa-b863b0f0044f",
      "status": "Failed",
      "startTime": "2026-10-02T18:49:07.875106668Z",
      "endTime": "2026-10-02T19:08:13.588598197Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-f65j4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-g5bl8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-f65j4: Creating: virtualmachine creating or updating\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-g5bl8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kfvn25" for cluster "cluster-candidate-4-21-kfvn25"
Unexpected error:
    <*fmt.wrapError | 0xc000b3b220>: 
    failed to create NodePool np-kfvn25: failed waiting for nodepool="np-kfvn25" for cluster "cluster-candidate-4-21-kfvn25" in resourcegroup="rg-candidate-4-21-85wnrgqqbd2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b12e98e2-7f93-4004-87fa-b863b0f0044f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b12e98e2-7f93-4004-87fa-b863b0f0044f",
      "name": "b12e98e2-7f93-4004-87fa-b863b0f0044f",
      "status": "Failed",
      "startTime": "2026-10-02T18:49:07.875106668Z",
      "endTime": "2026-10-02T19:08:13.588598197Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-f65j4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-g5bl8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-f65j4: Creating: virtualmachine creating or updating\nMachine l6p5f7e6i7t3n7m-np-kfvn25-4sz9p-g5bl8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vvnbz4" for cluster "cluster-candidate-4-20-vvnbz4"
Unexpected error:
    <*fmt.wrapError | 0xc000c04c80>: 
    failed to create NodePool np-vvnbz4: failed waiting for nodepool="np-vvnbz4" for cluster "cluster-candidate-4-20-vvnbz4" in resourcegroup="rg-candidate-4-20-6rrzp6smfk8r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a84f10bd-1000-4ff5-b696-ec0471f172d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a84f10bd-1000-4ff5-b696-ec0471f172d5",
      "name": "a84f10bd-1000-4ff5-b696-ec0471f172d5",
      "status": "Failed",
      "startTime": "2026-10-02T18:45:19.007012696Z",
      "endTime": "2026-10-02T19:04:23.174354297Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-jqrxn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-rd5xk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-jqrxn: Creating: virtualmachine creating or updating\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-rd5xk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vvnbz4" for cluster "cluster-candidate-4-20-vvnbz4"
Unexpected error:
    <*fmt.wrapError | 0xc000c04c80>: 
    failed to create NodePool np-vvnbz4: failed waiting for nodepool="np-vvnbz4" for cluster "cluster-candidate-4-20-vvnbz4" in resourcegroup="rg-candidate-4-20-6rrzp6smfk8r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a84f10bd-1000-4ff5-b696-ec0471f172d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a84f10bd-1000-4ff5-b696-ec0471f172d5",
      "name": "a84f10bd-1000-4ff5-b696-ec0471f172d5",
      "status": "Failed",
      "startTime": "2026-10-02T18:45:19.007012696Z",
      "endTime": "2026-10-02T19:04:23.174354297Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-jqrxn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-rd5xk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-jqrxn: Creating: virtualmachine creating or updating\nMachine j3l2b9y9v8f1o9h-np-vvnbz4-llscx-rd5xk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qv99kb" for cluster "cluster-candidate-4-23-qv99kb"
Unexpected error:
    <*fmt.wrapError | 0xc000c78ba0>: 
    failed to create NodePool np-qv99kb: failed waiting for nodepool="np-qv99kb" for cluster "cluster-candidate-4-23-qv99kb" in resourcegroup="rg-candidate-4-23-xhtfbmhw8nkt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93ec2ac0-8140-4df7-861f-8f5e69bd2825
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93ec2ac0-8140-4df7-861f-8f5e69bd2825",
      "name": "93ec2ac0-8140-4df7-861f-8f5e69bd2825",
      "status": "Failed",
      "startTime": "2026-10-02T18:47:56.273213854Z",
      "endTime": "2026-10-02T19:07:03.234946252Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-nhvhz: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-xq4s5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-nhvhz: Creating: virtualmachine creating or updating\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-xq4s5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qv99kb" for cluster "cluster-candidate-4-23-qv99kb"
Unexpected error:
    <*fmt.wrapError | 0xc000c78ba0>: 
    failed to create NodePool np-qv99kb: failed waiting for nodepool="np-qv99kb" for cluster "cluster-candidate-4-23-qv99kb" in resourcegroup="rg-candidate-4-23-xhtfbmhw8nkt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93ec2ac0-8140-4df7-861f-8f5e69bd2825
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/93ec2ac0-8140-4df7-861f-8f5e69bd2825",
      "name": "93ec2ac0-8140-4df7-861f-8f5e69bd2825",
      "status": "Failed",
      "startTime": "2026-10-02T18:47:56.273213854Z",
      "endTime": "2026-10-02T19:07:03.234946252Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-nhvhz: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-xq4s5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-nhvhz: Creating: virtualmachine creating or updating\nMachine o0z3a3e4t8t4l0z-np-qv99kb-rmmkl-xq4s5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xf2nmx" for cluster "cluster-candidate-5-0-xf2nmx"
Unexpected error:
    <*fmt.wrapError | 0xc000995320>: 
    failed to create NodePool np-xf2nmx: failed waiting for nodepool="np-xf2nmx" for cluster "cluster-candidate-5-0-xf2nmx" in resourcegroup="rg-candidate-5-0-7jr6wdpsfvz4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/992a0c66-9c4a-4dc5-8b0f-2c46d65526fe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/992a0c66-9c4a-4dc5-8b0f-2c46d65526fe",
      "name": "992a0c66-9c4a-4dc5-8b0f-2c46d65526fe",
      "status": "Failed",
      "startTime": "2026-10-02T18:49:51.032351879Z",
      "endTime": "2026-10-02T19:08:53.234722812Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-kqt4r: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-nwmrg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-kqt4r: Creating: virtualmachine creating or updating\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-nwmrg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xf2nmx" for cluster "cluster-candidate-5-0-xf2nmx"
Unexpected error:
    <*fmt.wrapError | 0xc000995320>: 
    failed to create NodePool np-xf2nmx: failed waiting for nodepool="np-xf2nmx" for cluster "cluster-candidate-5-0-xf2nmx" in resourcegroup="rg-candidate-5-0-7jr6wdpsfvz4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/992a0c66-9c4a-4dc5-8b0f-2c46d65526fe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/992a0c66-9c4a-4dc5-8b0f-2c46d65526fe",
      "name": "992a0c66-9c4a-4dc5-8b0f-2c46d65526fe",
      "status": "Failed",
      "startTime": "2026-10-02T18:49:51.032351879Z",
      "endTime": "2026-10-02T19:08:53.234722812Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-kqt4r: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-nwmrg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-kqt4r: Creating: virtualmachine creating or updating\nMachine b1t7y5a4d6l6s5p-np-xf2nmx-bnzwr-nwmrg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fcce00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-kndcxqxwg5m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9940d584-67e5-4b2a-939b-305a5a782054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9940d584-67e5-4b2a-939b-305a5a782054",
      "name": "9940d584-67e5-4b2a-939b-305a5a782054",
      "status": "Failed",
      "startTime": "2026-10-02T18:26:54.435762769Z",
      "endTime": "2026-10-02T18:46:02.862827665Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-6stxx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-fvklg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-6stxx: Creating: virtualmachine creating or updating\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-fvklg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fcce00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-kndcxqxwg5m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9940d584-67e5-4b2a-939b-305a5a782054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9940d584-67e5-4b2a-939b-305a5a782054",
      "name": "9940d584-67e5-4b2a-939b-305a5a782054",
      "status": "Failed",
      "startTime": "2026-10-02T18:26:54.435762769Z",
      "endTime": "2026-10-02T18:46:02.862827665Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-6stxx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-fvklg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-6stxx: Creating: virtualmachine creating or updating\nMachine l1e8j9h2p9p7t8t-np-1-dsx99-fvklg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lvw97g" for cluster "cluster-candidate-5-1-lvw97g"
Unexpected error:
    <*fmt.wrapError | 0xc000e66f00>: 
    failed to create NodePool np-lvw97g: failed waiting for nodepool="np-lvw97g" for cluster "cluster-candidate-5-1-lvw97g" in resourcegroup="rg-candidate-5-1-qrgrjlf6rkm9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a97c7bb-57af-45b7-8fb7-5b981cf7570f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a97c7bb-57af-45b7-8fb7-5b981cf7570f",
      "name": "4a97c7bb-57af-45b7-8fb7-5b981cf7570f",
      "status": "Failed",
      "startTime": "2026-10-02T18:50:51.431353811Z",
      "endTime": "2026-10-02T19:09:53.217243745Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-95ljv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-cfkc9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-95ljv: Creating: virtualmachine creating or updating\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-cfkc9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lvw97g" for cluster "cluster-candidate-5-1-lvw97g"
Unexpected error:
    <*fmt.wrapError | 0xc000e66f00>: 
    failed to create NodePool np-lvw97g: failed waiting for nodepool="np-lvw97g" for cluster "cluster-candidate-5-1-lvw97g" in resourcegroup="rg-candidate-5-1-qrgrjlf6rkm9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a97c7bb-57af-45b7-8fb7-5b981cf7570f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a97c7bb-57af-45b7-8fb7-5b981cf7570f",
      "name": "4a97c7bb-57af-45b7-8fb7-5b981cf7570f",
      "status": "Failed",
      "startTime": "2026-10-02T18:50:51.431353811Z",
      "endTime": "2026-10-02T19:09:53.217243745Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-95ljv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-cfkc9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-95ljv: Creating: virtualmachine creating or updating\nMachine l3n7x5q5h0o7l6w-np-lvw97g-9smn8-cfkc9: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000c9c280>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-gg8s9j" in resourcegroup="rg-np-downgrade-gg8s9j-fnkpnt9xd47t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4186f06-3e4f-4125-8780-405aa920bd3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4186f06-3e4f-4125-8780-405aa920bd3f",
      "name": "a4186f06-3e4f-4125-8780-405aa920bd3f",
      "status": "Failed",
      "startTime": "2026-10-02T19:29:21.050247186Z",
      "endTime": "2026-10-02T19:48:24.533281016Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-gg8s9j-fnkpnt9xd47t-np-dg-gg8s9j--managed/providers/Microsoft.Compute/virtualMachines/a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-gg8s9j-fnkpnt9xd47t-np-dg-gg8s9j--managed/providers/Microsoft.Compute/virtualMachines/a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000c9c280>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-gg8s9j" in resourcegroup="rg-np-downgrade-gg8s9j-fnkpnt9xd47t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4186f06-3e4f-4125-8780-405aa920bd3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4186f06-3e4f-4125-8780-405aa920bd3f",
      "name": "a4186f06-3e4f-4125-8780-405aa920bd3f",
      "status": "Failed",
      "startTime": "2026-10-02T19:29:21.050247186Z",
      "endTime": "2026-10-02T19:48:24.533281016Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-gg8s9j-fnkpnt9xd47t-np-dg-gg8s9j--managed/providers/Microsoft.Compute/virtualMachines/a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-gg8s9j-fnkpnt9xd47t-np-dg-gg8s9j--managed/providers/Microsoft.Compute/virtualMachines/a7o8k0t2r9e8d6s-npdg-4-21-xzxrt-5htvd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc001085220>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-dk96qxfcglkz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a2c0d616-d3e1-4467-a097-edfb164f91f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a2c0d616-d3e1-4467-a097-edfb164f91f8",
      "name": "a2c0d616-d3e1-4467-a097-edfb164f91f8",
      "status": "Failed",
      "startTime": "2026-10-02T18:26:30.147930745Z",
      "endTime": "2026-10-02T18:45:32.745630715Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-jbbww-kglw7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-jbbww-kglw7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc001085220>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-dk96qxfcglkz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a2c0d616-d3e1-4467-a097-edfb164f91f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a2c0d616-d3e1-4467-a097-edfb164f91f8",
      "name": "a2c0d616-d3e1-4467-a097-edfb164f91f8",
      "status": "Failed",
      "startTime": "2026-10-02T18:26:30.147930745Z",
      "endTime": "2026-10-02T18:45:32.745630715Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-jbbww-kglw7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-jbbww-kglw7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00341d8a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-8s9pgvgwlgbj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4e9a75c-b715-4a5f-8f78-888a4044e76e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4e9a75c-b715-4a5f-8f78-888a4044e76e",
      "name": "e4e9a75c-b715-4a5f-8f78-888a4044e76e",
      "status": "Failed",
      "startTime": "2026-10-02T19:20:19.695876004Z",
      "endTime": "2026-10-02T19:39:24.271020967Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-4mb96: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-8f5lf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-4mb96: Creating: virtualmachine creating or updating\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-8f5lf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00341d8a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-8s9pgvgwlgbj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4e9a75c-b715-4a5f-8f78-888a4044e76e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4e9a75c-b715-4a5f-8f78-888a4044e76e",
      "name": "e4e9a75c-b715-4a5f-8f78-888a4044e76e",
      "status": "Failed",
      "startTime": "2026-10-02T19:20:19.695876004Z",
      "endTime": "2026-10-02T19:39:24.271020967Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-4mb96: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-8f5lf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-4mb96: Creating: virtualmachine creating or updating\nMachine k8i8k9n2m6a4a0n-np-delete-main-pkm89-8f5lf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f12060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-5x984bppgxmk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f889fa2b-1853-4b53-b0a5-8af8bb1664c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f889fa2b-1853-4b53-b0a5-8af8bb1664c4",
      "name": "f889fa2b-1853-4b53-b0a5-8af8bb1664c4",
      "status": "Failed",
      "startTime": "2026-10-02T18:28:21.353402223Z",
      "endTime": "2026-10-02T18:47:22.523072731Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-rvbvq-47m2n: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-rvbvq-dncqc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-rvbvq-47m2n: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-rvbvq-dncqc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f12060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-5x984bppgxmk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f889fa2b-1853-4b53-b0a5-8af8bb1664c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f889fa2b-1853-4b53-b0a5-8af8bb1664c4",
      "name": "f889fa2b-1853-4b53-b0a5-8af8bb1664c4",
      "status": "Failed",
      "startTime": "2026-10-02T18:28:21.353402223Z",
      "endTime": "2026-10-02T18:47:22.523072731Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-rvbvq-47m2n: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-rvbvq-dncqc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-rvbvq-47m2n: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-rvbvq-dncqc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000a9d580>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-vrplznt5v54r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85d24adc-6f21-47c1-8812-f58f8001bb49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85d24adc-6f21-47c1-8812-f58f8001bb49",
      "name": "85d24adc-6f21-47c1-8812-f58f8001bb49",
      "status": "Failed",
      "startTime": "2026-10-02T07:56:13.127412921Z",
      "endTime": "2026-10-02T08:15:18.247774488Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-t8dkq-6vlv4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-t8dkq-6vlv4: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000a9d580>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-vrplznt5v54r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85d24adc-6f21-47c1-8812-f58f8001bb49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85d24adc-6f21-47c1-8812-f58f8001bb49",
      "name": "85d24adc-6f21-47c1-8812-f58f8001bb49",
      "status": "Failed",
      "startTime": "2026-10-02T07:56:13.127412921Z",
      "endTime": "2026-10-02T08:15:18.247774488Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-t8dkq-6vlv4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-t8dkq-6vlv4: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000666c60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-bgj7nclctwxw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/29fe846e-0932-41fe-a2dd-ff17d6cc5ac0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/29fe846e-0932-41fe-a2dd-ff17d6cc5ac0",
      "name": "29fe846e-0932-41fe-a2dd-ff17d6cc5ac0",
      "status": "Failed",
      "startTime": "2026-10-02T08:52:01.290279416Z",
      "endTime": "2026-10-02T09:11:09.436811147Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-4k649: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bgj7nclctwxw--managed/providers/Microsoft.Compute/virtualMachines/h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-4k649: Creating: virtualmachine creating or updating\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bgj7nclctwxw--managed/providers/Microsoft.Compute/virtualMachines/h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000666c60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-bgj7nclctwxw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/29fe846e-0932-41fe-a2dd-ff17d6cc5ac0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/29fe846e-0932-41fe-a2dd-ff17d6cc5ac0",
      "name": "29fe846e-0932-41fe-a2dd-ff17d6cc5ac0",
      "status": "Failed",
      "startTime": "2026-10-02T08:52:01.290279416Z",
      "endTime": "2026-10-02T09:11:09.436811147Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-4k649: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bgj7nclctwxw--managed/providers/Microsoft.Compute/virtualMachines/h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-4k649: Creating: virtualmachine creating or updating\nMachine h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bgj7nclctwxw--managed/providers/Microsoft.Compute/virtualMachines/h3r3f2u9k1p8i1c-np-delete-main-hwgw9-p69q6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-q645ss" for cluster "cluster-candidate-4-23-q645ss"
Unexpected error:
    <*fmt.wrapError | 0xc00147edc0>: 
    failed to create NodePool np-q645ss: failed waiting for nodepool="np-q645ss" for cluster "cluster-candidate-4-23-q645ss" in resourcegroup="rg-candidate-4-23-97lbmpnc95ml" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7c6ae1c-be71-48cb-b2ba-de56bb6c16d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7c6ae1c-be71-48cb-b2ba-de56bb6c16d5",
      "name": "e7c6ae1c-be71-48cb-b2ba-de56bb6c16d5",
      "status": "Failed",
      "startTime": "2026-10-02T08:22:25.175276447Z",
      "endTime": "2026-10-02T08:41:25.163899948Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-b9h5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-rcd4l: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-b9h5q: Creating: virtualmachine creating or updating\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-rcd4l: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-q645ss" for cluster "cluster-candidate-4-23-q645ss"
Unexpected error:
    <*fmt.wrapError | 0xc00147edc0>: 
    failed to create NodePool np-q645ss: failed waiting for nodepool="np-q645ss" for cluster "cluster-candidate-4-23-q645ss" in resourcegroup="rg-candidate-4-23-97lbmpnc95ml" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7c6ae1c-be71-48cb-b2ba-de56bb6c16d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e7c6ae1c-be71-48cb-b2ba-de56bb6c16d5",
      "name": "e7c6ae1c-be71-48cb-b2ba-de56bb6c16d5",
      "status": "Failed",
      "startTime": "2026-10-02T08:22:25.175276447Z",
      "endTime": "2026-10-02T08:41:25.163899948Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-b9h5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-rcd4l: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-b9h5q: Creating: virtualmachine creating or updating\nMachine x4b4k7o1x7w4m8z-np-q645ss-zvxb8-rcd4l: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ea0360>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5w2whf62mkf6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffd107ed-298a-4008-9f25-801165f1c37e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffd107ed-298a-4008-9f25-801165f1c37e",
      "name": "ffd107ed-298a-4008-9f25-801165f1c37e",
      "status": "Failed",
      "startTime": "2026-10-02T07:56:52.717139839Z",
      "endTime": "2026-10-02T08:15:58.568270167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5w2whf62mkf6--managed/providers/Microsoft.Compute/virtualMachines/v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5w2whf62mkf6--managed/providers/Microsoft.Compute/virtualMachines/v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ea0360>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5w2whf62mkf6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffd107ed-298a-4008-9f25-801165f1c37e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffd107ed-298a-4008-9f25-801165f1c37e",
      "name": "ffd107ed-298a-4008-9f25-801165f1c37e",
      "status": "Failed",
      "startTime": "2026-10-02T07:56:52.717139839Z",
      "endTime": "2026-10-02T08:15:58.568270167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5w2whf62mkf6--managed/providers/Microsoft.Compute/virtualMachines/v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5w2whf62mkf6--managed/providers/Microsoft.Compute/virtualMachines/v5b2u0b6b5e2c6j-np-1-rtdvc-2drvl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kf44kx" for cluster "cluster-candidate-4-22-kf44kx"
Unexpected error:
    <*fmt.wrapError | 0xc001670be0>: 
    failed to create NodePool np-kf44kx: failed waiting for nodepool="np-kf44kx" for cluster "cluster-candidate-4-22-kf44kx" in resourcegroup="rg-candidate-4-22-p972hm7v767t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5cc41ef-2098-4080-80d0-52d4a6c605b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5cc41ef-2098-4080-80d0-52d4a6c605b2",
      "name": "f5cc41ef-2098-4080-80d0-52d4a6c605b2",
      "status": "Failed",
      "startTime": "2026-10-02T08:24:18.844576711Z",
      "endTime": "2026-10-02T08:43:25.00034882Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-ff2wq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-p7lf6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-ff2wq: Creating: virtualmachine creating or updating\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-p7lf6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kf44kx" for cluster "cluster-candidate-4-22-kf44kx"
Unexpected error:
    <*fmt.wrapError | 0xc001670be0>: 
    failed to create NodePool np-kf44kx: failed waiting for nodepool="np-kf44kx" for cluster "cluster-candidate-4-22-kf44kx" in resourcegroup="rg-candidate-4-22-p972hm7v767t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5cc41ef-2098-4080-80d0-52d4a6c605b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5cc41ef-2098-4080-80d0-52d4a6c605b2",
      "name": "f5cc41ef-2098-4080-80d0-52d4a6c605b2",
      "status": "Failed",
      "startTime": "2026-10-02T08:24:18.844576711Z",
      "endTime": "2026-10-02T08:43:25.00034882Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-ff2wq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-p7lf6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-ff2wq: Creating: virtualmachine creating or updating\nMachine j5o2i4r2x6i1i6v-np-kf44kx-5lnfn-p7lf6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qbwjq7" for cluster "cluster-candidate-5-1-qbwjq7"
Unexpected error:
    <*fmt.wrapError | 0xc000f28640>: 
    failed to create NodePool np-qbwjq7: failed waiting for nodepool="np-qbwjq7" for cluster "cluster-candidate-5-1-qbwjq7" in resourcegroup="rg-candidate-5-1-cxt8gqtcm56h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/411adf6f-822a-4f98-bc7d-64851f61532c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/411adf6f-822a-4f98-bc7d-64851f61532c",
      "name": "411adf6f-822a-4f98-bc7d-64851f61532c",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:47.400908917Z",
      "endTime": "2026-10-02T08:42:55.012095583Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-dhvsm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-zlljx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-dhvsm: Creating: virtualmachine creating or updating\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-zlljx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qbwjq7" for cluster "cluster-candidate-5-1-qbwjq7"
Unexpected error:
    <*fmt.wrapError | 0xc000f28640>: 
    failed to create NodePool np-qbwjq7: failed waiting for nodepool="np-qbwjq7" for cluster "cluster-candidate-5-1-qbwjq7" in resourcegroup="rg-candidate-5-1-cxt8gqtcm56h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/411adf6f-822a-4f98-bc7d-64851f61532c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/411adf6f-822a-4f98-bc7d-64851f61532c",
      "name": "411adf6f-822a-4f98-bc7d-64851f61532c",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:47.400908917Z",
      "endTime": "2026-10-02T08:42:55.012095583Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-dhvsm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-zlljx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-dhvsm: Creating: virtualmachine creating or updating\nMachine x5o3w3y8k8n9n2p-np-qbwjq7-vv8c8-zlljx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc000e761b0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-dwl8n7k5d292" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cd68833-1cd8-4477-9f36-56a0fe7d00ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cd68833-1cd8-4477-9f36-56a0fe7d00ba",
      "name": "0cd68833-1cd8-4477-9f36-56a0fe7d00ba",
      "status": "Failed",
      "startTime": "2026-10-02T07:58:36.615555909Z",
      "endTime": "2026-10-02T08:17:38.532193005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-c9phq-b96vq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-c9phq-gw2rm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-c9phq-b96vq: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-c9phq-gw2rm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc000e761b0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-dwl8n7k5d292" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cd68833-1cd8-4477-9f36-56a0fe7d00ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0cd68833-1cd8-4477-9f36-56a0fe7d00ba",
      "name": "0cd68833-1cd8-4477-9f36-56a0fe7d00ba",
      "status": "Failed",
      "startTime": "2026-10-02T07:58:36.615555909Z",
      "endTime": "2026-10-02T08:17:38.532193005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-c9phq-b96vq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-c9phq-gw2rm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-c9phq-b96vq: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-c9phq-gw2rm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0008aece0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-plwp75qbhvcq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d68804b-aa1f-441d-9c44-b697ee5be0e1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d68804b-aa1f-441d-9c44-b697ee5be0e1",
      "name": "0d68804b-aa1f-441d-9c44-b697ee5be0e1",
      "status": "Failed",
      "startTime": "2026-10-02T07:58:27.095320904Z",
      "endTime": "2026-10-02T08:17:28.456252233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-j6stv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-jcmz5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-j6stv: Creating: virtualmachine creating or updating\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-jcmz5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0008aece0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-plwp75qbhvcq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d68804b-aa1f-441d-9c44-b697ee5be0e1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d68804b-aa1f-441d-9c44-b697ee5be0e1",
      "name": "0d68804b-aa1f-441d-9c44-b697ee5be0e1",
      "status": "Failed",
      "startTime": "2026-10-02T07:58:27.095320904Z",
      "endTime": "2026-10-02T08:17:28.456252233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-j6stv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-jcmz5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-j6stv: Creating: virtualmachine creating or updating\nMachine v5t1z8u9w4x9o4c-np-1-qj5gs-jcmz5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-65j846" for cluster "cluster-candidate-4-21-65j846"
Unexpected error:
    <*fmt.wrapError | 0xc000dd8540>: 
    failed to create NodePool np-65j846: failed waiting for nodepool="np-65j846" for cluster "cluster-candidate-4-21-65j846" in resourcegroup="rg-candidate-4-21-8dj7spdjvcps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4b395e6-2b88-4e54-814b-bcbf478cdffc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4b395e6-2b88-4e54-814b-bcbf478cdffc",
      "name": "d4b395e6-2b88-4e54-814b-bcbf478cdffc",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:11.897579976Z",
      "endTime": "2026-10-02T08:42:15.234899325Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-dczz6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-hp4tn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-dczz6: Creating: virtualmachine creating or updating\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-hp4tn: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-65j846" for cluster "cluster-candidate-4-21-65j846"
Unexpected error:
    <*fmt.wrapError | 0xc000dd8540>: 
    failed to create NodePool np-65j846: failed waiting for nodepool="np-65j846" for cluster "cluster-candidate-4-21-65j846" in resourcegroup="rg-candidate-4-21-8dj7spdjvcps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4b395e6-2b88-4e54-814b-bcbf478cdffc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4b395e6-2b88-4e54-814b-bcbf478cdffc",
      "name": "d4b395e6-2b88-4e54-814b-bcbf478cdffc",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:11.897579976Z",
      "endTime": "2026-10-02T08:42:15.234899325Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-dczz6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-hp4tn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-dczz6: Creating: virtualmachine creating or updating\nMachine p4r1l1y9v2e2n2b-np-65j846-2t8dj-hp4tn: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002194198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kkklwf69d6rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6596056-a431-4061-9fcc-922385eb88b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6596056-a431-4061-9fcc-922385eb88b7",
      "name": "f6596056-a431-4061-9fcc-922385eb88b7",
      "status": "Failed",
      "startTime": "2026-10-02T07:58:26.598790316Z",
      "endTime": "2026-10-02T08:17:28.2532697Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-t9npj-5lpcd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-t9npj-z8x9w: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-t9npj-5lpcd: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-t9npj-z8x9w: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002194198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kkklwf69d6rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6596056-a431-4061-9fcc-922385eb88b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f6596056-a431-4061-9fcc-922385eb88b7",
      "name": "f6596056-a431-4061-9fcc-922385eb88b7",
      "status": "Failed",
      "startTime": "2026-10-02T07:58:26.598790316Z",
      "endTime": "2026-10-02T08:17:28.2532697Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-t9npj-5lpcd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-t9npj-z8x9w: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-t9npj-5lpcd: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-t9npj-z8x9w: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-96gwgz" for cluster "cluster-candidate-5-0-96gwgz"
Unexpected error:
    <*fmt.wrapError | 0xc000cf2260>: 
    failed to create NodePool np-96gwgz: failed waiting for nodepool="np-96gwgz" for cluster "cluster-candidate-5-0-96gwgz" in resourcegroup="rg-candidate-5-0-84kwhzj8r7w4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7df18c5-1f74-4f58-93d8-951ddd66b332
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7df18c5-1f74-4f58-93d8-951ddd66b332",
      "name": "c7df18c5-1f74-4f58-93d8-951ddd66b332",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:19.198498666Z",
      "endTime": "2026-10-02T08:42:25.18491611Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-4frj5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-d5fmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-4frj5: Creating: virtualmachine creating or updating\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-d5fmk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-96gwgz" for cluster "cluster-candidate-5-0-96gwgz"
Unexpected error:
    <*fmt.wrapError | 0xc000cf2260>: 
    failed to create NodePool np-96gwgz: failed waiting for nodepool="np-96gwgz" for cluster "cluster-candidate-5-0-96gwgz" in resourcegroup="rg-candidate-5-0-84kwhzj8r7w4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7df18c5-1f74-4f58-93d8-951ddd66b332
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7df18c5-1f74-4f58-93d8-951ddd66b332",
      "name": "c7df18c5-1f74-4f58-93d8-951ddd66b332",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:19.198498666Z",
      "endTime": "2026-10-02T08:42:25.18491611Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-4frj5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-d5fmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-4frj5: Creating: virtualmachine creating or updating\nMachine m9p3c0m3i2a9q0y-np-96gwgz-x5zk7-d5fmk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tgr2wf" for cluster "cluster-candidate-4-20-tgr2wf"
Unexpected error:
    <*fmt.wrapError | 0xc000e1d940>: 
    failed to create NodePool np-tgr2wf: failed waiting for nodepool="np-tgr2wf" for cluster "cluster-candidate-4-20-tgr2wf" in resourcegroup="rg-candidate-4-20-r2lrfp2tbsqc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0734784e-738a-4b64-ad9c-e373ea6e9682
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0734784e-738a-4b64-ad9c-e373ea6e9682",
      "name": "0734784e-738a-4b64-ad9c-e373ea6e9682",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:20.483388473Z",
      "endTime": "2026-10-02T08:42:25.166235902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-jmm8m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-k7sxl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-jmm8m: Creating: virtualmachine creating or updating\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-k7sxl: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tgr2wf" for cluster "cluster-candidate-4-20-tgr2wf"
Unexpected error:
    <*fmt.wrapError | 0xc000e1d940>: 
    failed to create NodePool np-tgr2wf: failed waiting for nodepool="np-tgr2wf" for cluster "cluster-candidate-4-20-tgr2wf" in resourcegroup="rg-candidate-4-20-r2lrfp2tbsqc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0734784e-738a-4b64-ad9c-e373ea6e9682
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0734784e-738a-4b64-ad9c-e373ea6e9682",
      "name": "0734784e-738a-4b64-ad9c-e373ea6e9682",
      "status": "Failed",
      "startTime": "2026-10-02T08:23:20.483388473Z",
      "endTime": "2026-10-02T08:42:25.166235902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-jmm8m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-k7sxl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-jmm8m: Creating: virtualmachine creating or updating\nMachine x6z3k1g0g0y1m3q-np-tgr2wf-4vbp4-k7sxl: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001276700>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5rf62rf4p9wv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d53fa45c-a9c4-43a3-b87a-c16df786eeb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d53fa45c-a9c4-43a3-b87a-c16df786eeb4",
      "name": "d53fa45c-a9c4-43a3-b87a-c16df786eeb4",
      "status": "Failed",
      "startTime": "2026-10-02T04:09:42.171003862Z",
      "endTime": "2026-10-02T04:28:46.827990048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-n4l4p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-qtmzk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-n4l4p: Creating: virtualmachine creating or updating\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-qtmzk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001276700>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5rf62rf4p9wv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d53fa45c-a9c4-43a3-b87a-c16df786eeb4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d53fa45c-a9c4-43a3-b87a-c16df786eeb4",
      "name": "d53fa45c-a9c4-43a3-b87a-c16df786eeb4",
      "status": "Failed",
      "startTime": "2026-10-02T04:09:42.171003862Z",
      "endTime": "2026-10-02T04:28:46.827990048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-n4l4p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-qtmzk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-n4l4p: Creating: virtualmachine creating or updating\nMachine y8h6g1h7a1t1v0i-np-deletable-c6pz2-qtmzk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-9rfwql" for cluster "cluster-candidate-5-1-9rfwql"
Unexpected error:
    <*fmt.wrapError | 0xc001103040>: 
    failed to create NodePool np-9rfwql: failed waiting for nodepool="np-9rfwql" for cluster "cluster-candidate-5-1-9rfwql" in resourcegroup="rg-candidate-5-1-jznzltj2rxxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fd61fa-5104-45ad-bf92-0d32ab5a44e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fd61fa-5104-45ad-bf92-0d32ab5a44e5",
      "name": "34fd61fa-5104-45ad-bf92-0d32ab5a44e5",
      "status": "Failed",
      "startTime": "2026-10-02T03:44:17.773093204Z",
      "endTime": "2026-10-02T04:03:23.679415683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-9ptnl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-gscjf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-9ptnl: Creating: virtualmachine creating or updating\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-gscjf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-9rfwql" for cluster "cluster-candidate-5-1-9rfwql"
Unexpected error:
    <*fmt.wrapError | 0xc001103040>: 
    failed to create NodePool np-9rfwql: failed waiting for nodepool="np-9rfwql" for cluster "cluster-candidate-5-1-9rfwql" in resourcegroup="rg-candidate-5-1-jznzltj2rxxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fd61fa-5104-45ad-bf92-0d32ab5a44e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fd61fa-5104-45ad-bf92-0d32ab5a44e5",
      "name": "34fd61fa-5104-45ad-bf92-0d32ab5a44e5",
      "status": "Failed",
      "startTime": "2026-10-02T03:44:17.773093204Z",
      "endTime": "2026-10-02T04:03:23.679415683Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-9ptnl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-gscjf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-9ptnl: Creating: virtualmachine creating or updating\nMachine s2j2b3a4i7k6f9u-np-9rfwql-2xh25-gscjf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc002984078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-4httpsjkvppc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27c9b728-c34e-4afa-93f4-b5118810ce82
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27c9b728-c34e-4afa-93f4-b5118810ce82",
      "name": "27c9b728-c34e-4afa-93f4-b5118810ce82",
      "status": "Failed",
      "startTime": "2026-10-02T03:19:57.766414352Z",
      "endTime": "2026-10-02T03:39:03.235272547Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-lszrc-8znc6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-lszrc-nxxm5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-lszrc-8znc6: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-lszrc-nxxm5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc002984078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-4httpsjkvppc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27c9b728-c34e-4afa-93f4-b5118810ce82
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27c9b728-c34e-4afa-93f4-b5118810ce82",
      "name": "27c9b728-c34e-4afa-93f4-b5118810ce82",
      "status": "Failed",
      "startTime": "2026-10-02T03:19:57.766414352Z",
      "endTime": "2026-10-02T03:39:03.235272547Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-lszrc-8znc6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-lszrc-nxxm5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-lszrc-8znc6: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-lszrc-nxxm5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fs2tl4" for cluster "cluster-candidate-4-22-fs2tl4"
Unexpected error:
    <*fmt.wrapError | 0xc000dde140>: 
    failed to create NodePool np-fs2tl4: failed waiting for nodepool="np-fs2tl4" for cluster "cluster-candidate-4-22-fs2tl4" in resourcegroup="rg-candidate-4-22-jzw6chr4d7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/af47af07-70e2-47bd-9306-0c7f478d186e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/af47af07-70e2-47bd-9306-0c7f478d186e",
      "name": "af47af07-70e2-47bd-9306-0c7f478d186e",
      "status": "Failed",
      "startTime": "2026-10-02T03:39:14.039068534Z",
      "endTime": "2026-10-02T03:58:23.505666093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-hwrpc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-mh58m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-hwrpc: Creating: virtualmachine creating or updating\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-mh58m: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fs2tl4" for cluster "cluster-candidate-4-22-fs2tl4"
Unexpected error:
    <*fmt.wrapError | 0xc000dde140>: 
    failed to create NodePool np-fs2tl4: failed waiting for nodepool="np-fs2tl4" for cluster "cluster-candidate-4-22-fs2tl4" in resourcegroup="rg-candidate-4-22-jzw6chr4d7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/af47af07-70e2-47bd-9306-0c7f478d186e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/af47af07-70e2-47bd-9306-0c7f478d186e",
      "name": "af47af07-70e2-47bd-9306-0c7f478d186e",
      "status": "Failed",
      "startTime": "2026-10-02T03:39:14.039068534Z",
      "endTime": "2026-10-02T03:58:23.505666093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-hwrpc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-mh58m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-hwrpc: Creating: virtualmachine creating or updating\nMachine g1v3q3i3v9q4u8m-np-fs2tl4-ttfnq-mh58m: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000943f60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6sjr57crkxhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a2572ed-7c03-46c2-98bb-89a19d5f777a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a2572ed-7c03-46c2-98bb-89a19d5f777a",
      "name": "3a2572ed-7c03-46c2-98bb-89a19d5f777a",
      "status": "Failed",
      "startTime": "2026-10-02T03:19:47.10992898Z",
      "endTime": "2026-10-02T03:38:53.263922354Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-9k9hg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-fjdfs: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-9k9hg: Creating: virtualmachine creating or updating\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-fjdfs: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000943f60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6sjr57crkxhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a2572ed-7c03-46c2-98bb-89a19d5f777a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3a2572ed-7c03-46c2-98bb-89a19d5f777a",
      "name": "3a2572ed-7c03-46c2-98bb-89a19d5f777a",
      "status": "Failed",
      "startTime": "2026-10-02T03:19:47.10992898Z",
      "endTime": "2026-10-02T03:38:53.263922354Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-9k9hg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-fjdfs: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-9k9hg: Creating: virtualmachine creating or updating\nMachine d3m7o6a3u1p6z3i-np-1-vf5g4-fjdfs: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2rhqzm" for cluster "cluster-candidate-4-23-2rhqzm"
Unexpected error:
    <*fmt.wrapError | 0xc000f90900>: 
    failed to create NodePool np-2rhqzm: failed waiting for nodepool="np-2rhqzm" for cluster "cluster-candidate-4-23-2rhqzm" in resourcegroup="rg-candidate-4-23-hvftzsslsv5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83438069-4881-4a7c-99c3-04167cb9898c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83438069-4881-4a7c-99c3-04167cb9898c",
      "name": "83438069-4881-4a7c-99c3-04167cb9898c",
      "status": "Failed",
      "startTime": "2026-10-02T03:40:49.053626001Z",
      "endTime": "2026-10-02T03:59:53.321378994Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-jdv5b: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-r825r: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-jdv5b: Creating: virtualmachine creating or updating\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-r825r: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2rhqzm" for cluster "cluster-candidate-4-23-2rhqzm"
Unexpected error:
    <*fmt.wrapError | 0xc000f90900>: 
    failed to create NodePool np-2rhqzm: failed waiting for nodepool="np-2rhqzm" for cluster "cluster-candidate-4-23-2rhqzm" in resourcegroup="rg-candidate-4-23-hvftzsslsv5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83438069-4881-4a7c-99c3-04167cb9898c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83438069-4881-4a7c-99c3-04167cb9898c",
      "name": "83438069-4881-4a7c-99c3-04167cb9898c",
      "status": "Failed",
      "startTime": "2026-10-02T03:40:49.053626001Z",
      "endTime": "2026-10-02T03:59:53.321378994Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-jdv5b: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-r825r: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-jdv5b: Creating: virtualmachine creating or updating\nMachine o2x4w0t0l5t3o4o-np-2rhqzm-ghd9l-r825r: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lfrdtx" for cluster "cluster-candidate-4-21-lfrdtx"
Unexpected error:
    <*fmt.wrapError | 0xc000a86ae0>: 
    failed to create NodePool np-lfrdtx: failed waiting for nodepool="np-lfrdtx" for cluster "cluster-candidate-4-21-lfrdtx" in resourcegroup="rg-candidate-4-21-v7j8q5wv69rf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8487946-6d32-4b69-bbc8-793ffc91ce5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8487946-6d32-4b69-bbc8-793ffc91ce5f",
      "name": "a8487946-6d32-4b69-bbc8-793ffc91ce5f",
      "status": "Failed",
      "startTime": "2026-10-02T03:40:32.227275462Z",
      "endTime": "2026-10-02T03:59:33.308103816Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-58v7d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-5gbq8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-58v7d: Creating: virtualmachine creating or updating\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-5gbq8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lfrdtx" for cluster "cluster-candidate-4-21-lfrdtx"
Unexpected error:
    <*fmt.wrapError | 0xc000a86ae0>: 
    failed to create NodePool np-lfrdtx: failed waiting for nodepool="np-lfrdtx" for cluster "cluster-candidate-4-21-lfrdtx" in resourcegroup="rg-candidate-4-21-v7j8q5wv69rf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8487946-6d32-4b69-bbc8-793ffc91ce5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8487946-6d32-4b69-bbc8-793ffc91ce5f",
      "name": "a8487946-6d32-4b69-bbc8-793ffc91ce5f",
      "status": "Failed",
      "startTime": "2026-10-02T03:40:32.227275462Z",
      "endTime": "2026-10-02T03:59:33.308103816Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-58v7d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-5gbq8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-58v7d: Creating: virtualmachine creating or updating\nMachine z7f1i8t4o9f5w4c-np-lfrdtx-fqngr-5gbq8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-s4kd9s" for cluster "cluster-candidate-4-20-s4kd9s"
Unexpected error:
    <*fmt.wrapError | 0xc000b0b4a0>: 
    failed to create NodePool np-s4kd9s: failed waiting for nodepool="np-s4kd9s" for cluster "cluster-candidate-4-20-s4kd9s" in resourcegroup="rg-candidate-4-20-zlm28jv4jtcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e5e7aaa-eb3f-4153-b0cf-40dab5c32152
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e5e7aaa-eb3f-4153-b0cf-40dab5c32152",
      "name": "7e5e7aaa-eb3f-4153-b0cf-40dab5c32152",
      "status": "Failed",
      "startTime": "2026-10-02T03:38:55.251415601Z",
      "endTime": "2026-10-02T03:58:03.48228814Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-mn8wj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-st4n7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-mn8wj: Creating: virtualmachine creating or updating\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-st4n7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-s4kd9s" for cluster "cluster-candidate-4-20-s4kd9s"
Unexpected error:
    <*fmt.wrapError | 0xc000b0b4a0>: 
    failed to create NodePool np-s4kd9s: failed waiting for nodepool="np-s4kd9s" for cluster "cluster-candidate-4-20-s4kd9s" in resourcegroup="rg-candidate-4-20-zlm28jv4jtcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e5e7aaa-eb3f-4153-b0cf-40dab5c32152
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e5e7aaa-eb3f-4153-b0cf-40dab5c32152",
      "name": "7e5e7aaa-eb3f-4153-b0cf-40dab5c32152",
      "status": "Failed",
      "startTime": "2026-10-02T03:38:55.251415601Z",
      "endTime": "2026-10-02T03:58:03.48228814Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-mn8wj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-st4n7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-mn8wj: Creating: virtualmachine creating or updating\nMachine j1u6v8i1i7q3s1b-np-s4kd9s-nmqgx-st4n7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc00064c1e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-gs752pln4txj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d6e4c14-f5a6-4fd9-881e-2a42f87e3555
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d6e4c14-f5a6-4fd9-881e-2a42f87e3555",
      "name": "9d6e4c14-f5a6-4fd9-881e-2a42f87e3555",
      "status": "Failed",
      "startTime": "2026-10-02T03:18:04.846590038Z",
      "endTime": "2026-10-02T03:37:13.231851998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-5dbrz-khxf4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-5dbrz-khxf4: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc00064c1e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-gs752pln4txj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d6e4c14-f5a6-4fd9-881e-2a42f87e3555
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9d6e4c14-f5a6-4fd9-881e-2a42f87e3555",
      "name": "9d6e4c14-f5a6-4fd9-881e-2a42f87e3555",
      "status": "Failed",
      "startTime": "2026-10-02T03:18:04.846590038Z",
      "endTime": "2026-10-02T03:37:13.231851998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-5dbrz-khxf4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-5dbrz-khxf4: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0011d4048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rrrmbd9jmmdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d52358f-8309-4bc7-af9f-a5884f9307e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d52358f-8309-4bc7-af9f-a5884f9307e0",
      "name": "6d52358f-8309-4bc7-af9f-a5884f9307e0",
      "status": "Failed",
      "startTime": "2026-10-02T03:16:58.467567062Z",
      "endTime": "2026-10-02T03:36:03.226032745Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-sqrfn-pjdbq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-sqrfn-vtbbr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-sqrfn-pjdbq: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-sqrfn-vtbbr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0011d4048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rrrmbd9jmmdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d52358f-8309-4bc7-af9f-a5884f9307e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6d52358f-8309-4bc7-af9f-a5884f9307e0",
      "name": "6d52358f-8309-4bc7-af9f-a5884f9307e0",
      "status": "Failed",
      "startTime": "2026-10-02T03:16:58.467567062Z",
      "endTime": "2026-10-02T03:36:03.226032745Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-sqrfn-pjdbq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-sqrfn-vtbbr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-sqrfn-pjdbq: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-sqrfn-vtbbr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2pqt8t" for cluster "cluster-candidate-5-0-2pqt8t"
Unexpected error:
    <*fmt.wrapError | 0xc000e6c920>: 
    failed to create NodePool np-2pqt8t: failed waiting for nodepool="np-2pqt8t" for cluster "cluster-candidate-5-0-2pqt8t" in resourcegroup="rg-candidate-5-0-5cmhl28njm2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1671f2aa-76df-4172-89e0-297ac668f021
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1671f2aa-76df-4172-89e0-297ac668f021",
      "name": "1671f2aa-76df-4172-89e0-297ac668f021",
      "status": "Failed",
      "startTime": "2026-10-02T03:42:21.546109728Z",
      "endTime": "2026-10-02T04:01:23.351194166Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-8bg9m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-v5mdw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-8bg9m: Creating: virtualmachine creating or updating\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-v5mdw: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2pqt8t" for cluster "cluster-candidate-5-0-2pqt8t"
Unexpected error:
    <*fmt.wrapError | 0xc000e6c920>: 
    failed to create NodePool np-2pqt8t: failed waiting for nodepool="np-2pqt8t" for cluster "cluster-candidate-5-0-2pqt8t" in resourcegroup="rg-candidate-5-0-5cmhl28njm2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1671f2aa-76df-4172-89e0-297ac668f021
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1671f2aa-76df-4172-89e0-297ac668f021",
      "name": "1671f2aa-76df-4172-89e0-297ac668f021",
      "status": "Failed",
      "startTime": "2026-10-02T03:42:21.546109728Z",
      "endTime": "2026-10-02T04:01:23.351194166Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-8bg9m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-v5mdw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-8bg9m: Creating: virtualmachine creating or updating\nMachine w7u4t7y3s9c5g5m-np-2pqt8t-xxc4f-v5mdw: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002434018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9gqhrxhspmxj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2689a9-2918-4b62-b602-77629a36d12f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2689a9-2918-4b62-b602-77629a36d12f",
      "name": "1e2689a9-2918-4b62-b602-77629a36d12f",
      "status": "Failed",
      "startTime": "2026-10-02T01:12:51.230795814Z",
      "endTime": "2026-10-02T01:31:55.658354454Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zw5cd-7wmwm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-zw5cd-t5w4x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zw5cd-7wmwm: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-zw5cd-t5w4x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002434018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-9gqhrxhspmxj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2689a9-2918-4b62-b602-77629a36d12f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2689a9-2918-4b62-b602-77629a36d12f",
      "name": "1e2689a9-2918-4b62-b602-77629a36d12f",
      "status": "Failed",
      "startTime": "2026-10-02T01:12:51.230795814Z",
      "endTime": "2026-10-02T01:31:55.658354454Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-zw5cd-7wmwm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-zw5cd-t5w4x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-zw5cd-7wmwm: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-zw5cd-t5w4x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tgw4bw" for cluster "cluster-candidate-5-0-tgw4bw"
Unexpected error:
    <*fmt.wrapError | 0xc0013d7020>: 
    failed to create NodePool np-tgw4bw: failed waiting for nodepool="np-tgw4bw" for cluster "cluster-candidate-5-0-tgw4bw" in resourcegroup="rg-candidate-5-0-c5dplrlznljs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4804927-bde6-4d56-b2a2-861fb62752d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4804927-bde6-4d56-b2a2-861fb62752d1",
      "name": "e4804927-bde6-4d56-b2a2-861fb62752d1",
      "status": "Failed",
      "startTime": "2026-10-02T01:36:37.66754756Z",
      "endTime": "2026-10-02T01:55:45.209455595Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-h9x7v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-q4chj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-h9x7v: Creating: virtualmachine creating or updating\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-q4chj: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tgw4bw" for cluster "cluster-candidate-5-0-tgw4bw"
Unexpected error:
    <*fmt.wrapError | 0xc0013d7020>: 
    failed to create NodePool np-tgw4bw: failed waiting for nodepool="np-tgw4bw" for cluster "cluster-candidate-5-0-tgw4bw" in resourcegroup="rg-candidate-5-0-c5dplrlznljs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4804927-bde6-4d56-b2a2-861fb62752d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4804927-bde6-4d56-b2a2-861fb62752d1",
      "name": "e4804927-bde6-4d56-b2a2-861fb62752d1",
      "status": "Failed",
      "startTime": "2026-10-02T01:36:37.66754756Z",
      "endTime": "2026-10-02T01:55:45.209455595Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-h9x7v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-q4chj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-h9x7v: Creating: virtualmachine creating or updating\nMachine h0a6a6p2r2u4e6h-np-tgw4bw-s7jjh-q4chj: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00101d200>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2lqkfn8tvwwp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4098a97-bd38-463d-bee6-54c33fcd21ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4098a97-bd38-463d-bee6-54c33fcd21ea",
      "name": "f4098a97-bd38-463d-bee6-54c33fcd21ea",
      "status": "Failed",
      "startTime": "2026-10-02T01:12:48.719133172Z",
      "endTime": "2026-10-02T01:31:55.837797735Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-kdjwf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-q7z6x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-kdjwf: Creating: virtualmachine creating or updating\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-q7z6x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00101d200>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2lqkfn8tvwwp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4098a97-bd38-463d-bee6-54c33fcd21ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4098a97-bd38-463d-bee6-54c33fcd21ea",
      "name": "f4098a97-bd38-463d-bee6-54c33fcd21ea",
      "status": "Failed",
      "startTime": "2026-10-02T01:12:48.719133172Z",
      "endTime": "2026-10-02T01:31:55.837797735Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-kdjwf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-q7z6x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-kdjwf: Creating: virtualmachine creating or updating\nMachine b6g9d8r2h2w9x0t-np-1-qhfgz-q7z6x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2dtcq8" for cluster "cluster-candidate-4-20-2dtcq8"
Unexpected error:
    <*fmt.wrapError | 0xc0018500e0>: 
    failed to create NodePool np-2dtcq8: failed waiting for nodepool="np-2dtcq8" for cluster "cluster-candidate-4-20-2dtcq8" in resourcegroup="rg-candidate-4-20-wsgq7rfg4j9c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4f8b062f-d3f0-4efa-9aa5-be9e1c5218c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4f8b062f-d3f0-4efa-9aa5-be9e1c5218c2",
      "name": "4f8b062f-d3f0-4efa-9aa5-be9e1c5218c2",
      "status": "Failed",
      "startTime": "2026-10-02T01:35:03.016964221Z",
      "endTime": "2026-10-02T01:54:05.336017308Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2dtcq8" for cluster "cluster-candidate-4-20-2dtcq8"
Unexpected error:
    <*fmt.wrapError | 0xc0018500e0>: 
    failed to create NodePool np-2dtcq8: failed waiting for nodepool="np-2dtcq8" for cluster "cluster-candidate-4-20-2dtcq8" in resourcegroup="rg-candidate-4-20-wsgq7rfg4j9c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4f8b062f-d3f0-4efa-9aa5-be9e1c5218c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4f8b062f-d3f0-4efa-9aa5-be9e1c5218c2",
      "name": "4f8b062f-d3f0-4efa-9aa5-be9e1c5218c2",
      "status": "Failed",
      "startTime": "2026-10-02T01:35:03.016964221Z",
      "endTime": "2026-10-02T01:54:05.336017308Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-z9vgpc" for cluster "cluster-candidate-4-23-z9vgpc"
Unexpected error:
    <*fmt.wrapError | 0xc000d433c0>: 
    failed to create NodePool np-z9vgpc: failed waiting for nodepool="np-z9vgpc" for cluster "cluster-candidate-4-23-z9vgpc" in resourcegroup="rg-candidate-4-23-k2q8jrl7bhxd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5150081-901b-4236-a201-530c0a2b188a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5150081-901b-4236-a201-530c0a2b188a",
      "name": "a5150081-901b-4236-a201-530c0a2b188a",
      "status": "Failed",
      "startTime": "2026-10-02T01:35:33.904818612Z",
      "endTime": "2026-10-02T01:54:35.210883513Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-54zr9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-tfmvd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-54zr9: Creating: virtualmachine creating or updating\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-tfmvd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-z9vgpc" for cluster "cluster-candidate-4-23-z9vgpc"
Unexpected error:
    <*fmt.wrapError | 0xc000d433c0>: 
    failed to create NodePool np-z9vgpc: failed waiting for nodepool="np-z9vgpc" for cluster "cluster-candidate-4-23-z9vgpc" in resourcegroup="rg-candidate-4-23-k2q8jrl7bhxd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5150081-901b-4236-a201-530c0a2b188a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5150081-901b-4236-a201-530c0a2b188a",
      "name": "a5150081-901b-4236-a201-530c0a2b188a",
      "status": "Failed",
      "startTime": "2026-10-02T01:35:33.904818612Z",
      "endTime": "2026-10-02T01:54:35.210883513Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-54zr9: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-tfmvd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-54zr9: Creating: virtualmachine creating or updating\nMachine y0w7y2a8c7t4f6j-np-z9vgpc-4fpk5-tfmvd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ljll7c" for cluster "cluster-candidate-5-1-ljll7c"
Unexpected error:
    <*fmt.wrapError | 0xc002d92dc0>: 
    failed to create NodePool np-ljll7c: failed waiting for nodepool="np-ljll7c" for cluster "cluster-candidate-5-1-ljll7c" in resourcegroup="rg-candidate-5-1-m96hr99msv6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3472529a-993b-497a-a853-58314ee41fea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3472529a-993b-497a-a853-58314ee41fea",
      "name": "3472529a-993b-497a-a853-58314ee41fea",
      "status": "Failed",
      "startTime": "2026-10-02T01:39:28.8432233Z",
      "endTime": "2026-10-02T01:58:35.211672532Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ljll7c" for cluster "cluster-candidate-5-1-ljll7c"
Unexpected error:
    <*fmt.wrapError | 0xc002d92dc0>: 
    failed to create NodePool np-ljll7c: failed waiting for nodepool="np-ljll7c" for cluster "cluster-candidate-5-1-ljll7c" in resourcegroup="rg-candidate-5-1-m96hr99msv6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3472529a-993b-497a-a853-58314ee41fea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3472529a-993b-497a-a853-58314ee41fea",
      "name": "3472529a-993b-497a-a853-58314ee41fea",
      "status": "Failed",
      "startTime": "2026-10-02T01:39:28.8432233Z",
      "endTime": "2026-10-02T01:58:35.211672532Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000b95a80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-vp5rgzktgrqp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7950f938-55ee-4ce6-a5ca-480917c685bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7950f938-55ee-4ce6-a5ca-480917c685bf",
      "name": "7950f938-55ee-4ce6-a5ca-480917c685bf",
      "status": "Failed",
      "startTime": "2026-10-02T01:15:07.199174306Z",
      "endTime": "2026-10-02T01:34:15.817819834Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-vqbnm-zdcgw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-vqbnm-zdcgw: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000b95a80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-vp5rgzktgrqp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7950f938-55ee-4ce6-a5ca-480917c685bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7950f938-55ee-4ce6-a5ca-480917c685bf",
      "name": "7950f938-55ee-4ce6-a5ca-480917c685bf",
      "status": "Failed",
      "startTime": "2026-10-02T01:15:07.199174306Z",
      "endTime": "2026-10-02T01:34:15.817819834Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-vqbnm-zdcgw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-vqbnm-zdcgw: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pdsktt" for cluster "cluster-candidate-4-22-pdsktt"
Unexpected error:
    <*fmt.wrapError | 0xc0010d83a0>: 
    failed to create NodePool np-pdsktt: failed waiting for nodepool="np-pdsktt" for cluster "cluster-candidate-4-22-pdsktt" in resourcegroup="rg-candidate-4-22-5hpdwnfcpnnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/987287c4-95cd-422d-be6d-282705a06618
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/987287c4-95cd-422d-be6d-282705a06618",
      "name": "987287c4-95cd-422d-be6d-282705a06618",
      "status": "Failed",
      "startTime": "2026-10-02T01:35:11.708229062Z",
      "endTime": "2026-10-02T01:54:15.19502959Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-b9m75: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-vsxkb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-b9m75: Creating: virtualmachine creating or updating\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-vsxkb: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pdsktt" for cluster "cluster-candidate-4-22-pdsktt"
Unexpected error:
    <*fmt.wrapError | 0xc0010d83a0>: 
    failed to create NodePool np-pdsktt: failed waiting for nodepool="np-pdsktt" for cluster "cluster-candidate-4-22-pdsktt" in resourcegroup="rg-candidate-4-22-5hpdwnfcpnnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/987287c4-95cd-422d-be6d-282705a06618
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/987287c4-95cd-422d-be6d-282705a06618",
      "name": "987287c4-95cd-422d-be6d-282705a06618",
      "status": "Failed",
      "startTime": "2026-10-02T01:35:11.708229062Z",
      "endTime": "2026-10-02T01:54:15.19502959Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-b9m75: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-vsxkb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-b9m75: Creating: virtualmachine creating or updating\nMachine m9p9y7p7y8q8y0t-np-pdsktt-tcq2h-vsxkb: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001a9c060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v2ffkzvzwhp9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/59c86be8-df72-4988-a51b-6b1b20b915f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/59c86be8-df72-4988-a51b-6b1b20b915f4",
      "name": "59c86be8-df72-4988-a51b-6b1b20b915f4",
      "status": "Failed",
      "startTime": "2026-10-02T01:13:42.494621442Z",
      "endTime": "2026-10-02T01:32:45.646902114Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-69vhp-6nxhl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-69vhp-rch98: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-69vhp-6nxhl: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-69vhp-rch98: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001a9c060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v2ffkzvzwhp9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/59c86be8-df72-4988-a51b-6b1b20b915f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/59c86be8-df72-4988-a51b-6b1b20b915f4",
      "name": "59c86be8-df72-4988-a51b-6b1b20b915f4",
      "status": "Failed",
      "startTime": "2026-10-02T01:13:42.494621442Z",
      "endTime": "2026-10-02T01:32:45.646902114Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-69vhp-6nxhl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-69vhp-rch98: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-69vhp-6nxhl: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-69vhp-rch98: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tnjk89" for cluster "cluster-candidate-4-21-tnjk89"
Unexpected error:
    <*fmt.wrapError | 0xc000a24560>: 
    failed to create NodePool np-tnjk89: failed waiting for nodepool="np-tnjk89" for cluster "cluster-candidate-4-21-tnjk89" in resourcegroup="rg-candidate-4-21-hbwb4qtqm5m5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c80392bf-1395-461b-9b36-b3ca3dfa3732
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c80392bf-1395-461b-9b36-b3ca3dfa3732",
      "name": "c80392bf-1395-461b-9b36-b3ca3dfa3732",
      "status": "Failed",
      "startTime": "2026-10-02T01:34:52.67307417Z",
      "endTime": "2026-10-02T01:53:55.817268557Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-mfsnf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-rmqgp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-mfsnf: Creating: virtualmachine creating or updating\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-rmqgp: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tnjk89" for cluster "cluster-candidate-4-21-tnjk89"
Unexpected error:
    <*fmt.wrapError | 0xc000a24560>: 
    failed to create NodePool np-tnjk89: failed waiting for nodepool="np-tnjk89" for cluster "cluster-candidate-4-21-tnjk89" in resourcegroup="rg-candidate-4-21-hbwb4qtqm5m5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c80392bf-1395-461b-9b36-b3ca3dfa3732
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c80392bf-1395-461b-9b36-b3ca3dfa3732",
      "name": "c80392bf-1395-461b-9b36-b3ca3dfa3732",
      "status": "Failed",
      "startTime": "2026-10-02T01:34:52.67307417Z",
      "endTime": "2026-10-02T01:53:55.817268557Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-mfsnf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-rmqgp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-mfsnf: Creating: virtualmachine creating or updating\nMachine x5t3x3n0c0q1y5x-np-tnjk89-psj6d-rmqgp: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000eaa180>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ks2bs7m442f9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3917dbfd-2f9f-4a35-8346-d9c0b133e313
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3917dbfd-2f9f-4a35-8346-d9c0b133e313",
      "name": "3917dbfd-2f9f-4a35-8346-d9c0b133e313",
      "status": "Failed",
      "startTime": "2026-10-02T02:04:51.953995015Z",
      "endTime": "2026-10-02T02:23:58.43798257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-85c4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-g4qwf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-85c4g: Creating: virtualmachine creating or updating\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-g4qwf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000eaa180>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ks2bs7m442f9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3917dbfd-2f9f-4a35-8346-d9c0b133e313
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3917dbfd-2f9f-4a35-8346-d9c0b133e313",
      "name": "3917dbfd-2f9f-4a35-8346-d9c0b133e313",
      "status": "Failed",
      "startTime": "2026-10-02T02:04:51.953995015Z",
      "endTime": "2026-10-02T02:23:58.43798257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-85c4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-g4qwf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-85c4g: Creating: virtualmachine creating or updating\nMachine w5t1z3a1d2p4k6h-np-deletable-l2j9j-g4qwf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...