fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x152c481045e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s55djgjncgbr" to finish creating: GET https://rp.j5664384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x152c481045e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s55djgjncgbr" to finish creating: GET https://rp.j5664384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0xc0013d1ed0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/services-j7538560"
        }
    }
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0xc0013d1ed0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/services-j7538560"
        }
    }
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz"
Unexpected error:
    <*fmt.wrapError | 0xc001182600>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz" in resourcegroup="rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt" to finish creating: GET https://rp.j6364800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "name": "6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "status": "Failed",
      "startTime": "2026-10-03T00:01:24.493949107Z",
      "endTime": "2026-10-03T00:20:29.737570785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz"
Unexpected error:
    <*fmt.wrapError | 0xc001182600>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz" in resourcegroup="rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt" to finish creating: GET https://rp.j6364800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "name": "6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "status": "Failed",
      "startTime": "2026-10-03T00:01:24.493949107Z",
      "endTime": "2026-10-03T00:20:29.737570785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:06:25Z
  Ended:       2026-10-03T00:13:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6364800-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:07:24Z
  Ended:       2026-10-03T00:19:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7564928-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T00:08:27Z
  Ended:       2026-10-03T00:27:27Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7564928-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 9 time(s) (1 unknown, 8 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:22:54Z
  Ended:       2026-10-03T00:28:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7564928-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6vba5aukt21s79ka316m1hqn628hho-s4a5z1v1k7m8i6h", pod="openshift-apiserver-64ddcdd5c4-tjv4j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6vba5aukt21s79ka316m1hqn628hho-s4a5z1v1k7m8i6h/openshift-apiserver-64ddcdd5c4-tjv4j has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:19:29Z
  Ended:       2026-10-03T00:25:28Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9271680-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9271680-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-03T00:35:30Z
  Ended:       2026-10-03T00:41:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9271680-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9271680-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5fqc8n" on minor version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000b34440>: 
    failed waiting for hcpCluster="cluster-zstream-4-22-5fqc8n" in resourcegroup="rg-zstream-upgrade-4-22-tqtxbbsf75hl" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "name": "30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "status": "Failed",
      "startTime": "2026-10-02T23:01:14.974688957Z",
      "endTime": "2026-10-02T23:45:16.632504331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5fqc8n" on minor version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000b34440>: 
    failed waiting for hcpCluster="cluster-zstream-4-22-5fqc8n" in resourcegroup="rg-zstream-upgrade-4-22-tqtxbbsf75hl" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "name": "30b62d2f-e329-486f-a6aa-a4fc7c209017",
      "status": "Failed",
      "startTime": "2026-10-02T23:01:14.974688957Z",
      "endTime": "2026-10-02T23:45:16.632504331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fskm94" on minor version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a32b40>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-fskm94" in resourcegroup="rg-zstream-upgrade-4-20-7d7sxcc5nbp4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "name": "0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:38.87341489Z",
      "endTime": "2026-10-02T23:43:46.612931387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fskm94" on minor version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a32b40>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-fskm94" in resourcegroup="rg-zstream-upgrade-4-20-7d7sxcc5nbp4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "name": "0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:38.87341489Z",
      "endTime": "2026-10-02T23:43:46.612931387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-xxdkbk" on minor version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc00060c400>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-xxdkbk" in resourcegroup="rg-zstream-upgrade-4-21-fkvp7fgld7fx" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866",
      "name": "3e466c8d-b439-4e8d-9c16-270890d65866",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:19.732496702Z",
      "endTime": "2026-10-02T23:43:26.633598885Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-xxdkbk" on minor version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc00060c400>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-xxdkbk" in resourcegroup="rg-zstream-upgrade-4-21-fkvp7fgld7fx" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3e466c8d-b439-4e8d-9c16-270890d65866",
      "name": "3e466c8d-b439-4e8d-9c16-270890d65866",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:19.732496702Z",
      "endTime": "2026-10-02T23:43:26.633598885Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*fmt.wrapError | 0xc000a6f800>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-xzgkv9b984lz" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "name": "bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:51.980435313Z",
      "endTime": "2026-10-02T23:08:01.293538008Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*fmt.wrapError | 0xc000a6f800>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-xzgkv9b984lz" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "name": "bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:51.980435313Z",
      "endTime": "2026-10-02T23:08:01.293538008Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-qrrxbq" on minor version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000b7e340>: 
    failed waiting for hcpCluster="cluster-zstream-5-0-qrrxbq" in resourcegroup="rg-zstream-upgrade-5-0-bnn4f57kv77n" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626",
      "name": "c7191531-446c-4865-aac9-dd2569450626",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:13.4607593Z",
      "endTime": "2026-10-02T23:42:16.686387844Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-qrrxbq" on minor version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000b7e340>: 
    failed waiting for hcpCluster="cluster-zstream-5-0-qrrxbq" in resourcegroup="rg-zstream-upgrade-5-0-bnn4f57kv77n" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7191531-446c-4865-aac9-dd2569450626",
      "name": "c7191531-446c-4865-aac9-dd2569450626",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:13.4607593Z",
      "endTime": "2026-10-02T23:42:16.686387844Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-59v6sn" on minor version 4.23
Unexpected error:
    <*fmt.wrapError | 0xc000be1060>: 
    failed waiting for hcpCluster="cluster-zstream-4-23-59v6sn" in resourcegroup="rg-zstream-upgrade-4-23-dwdrgs9rpkt4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32",
      "name": "f2c90638-d31c-4819-aa00-1f73910cfe32",
      "status": "Failed",
      "startTime": "2026-10-02T23:00:29.412121269Z",
      "endTime": "2026-10-02T23:44:36.589759019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-59v6sn" on minor version 4.23
Unexpected error:
    <*fmt.wrapError | 0xc000be1060>: 
    failed waiting for hcpCluster="cluster-zstream-4-23-59v6sn" in resourcegroup="rg-zstream-upgrade-4-23-dwdrgs9rpkt4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32",
      "name": "f2c90638-d31c-4819-aa00-1f73910cfe32",
      "status": "Failed",
      "startTime": "2026-10-02T23:00:29.412121269Z",
      "endTime": "2026-10-02T23:44:36.589759019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ad2a60>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9h4twv" in resourcegroup="rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx" to finish creating: GET https://rp.j9301120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072",
      "name": "798eb1c5-8054-497c-b466-6bdb523be072",
      "status": "Failed",
      "startTime": "2026-10-02T23:16:07.311984652Z",
      "endTime": "2026-10-02T23:35:13.399526187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ad2a60>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9h4twv" in resourcegroup="rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx" to finish creating: GET https://rp.j9301120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072",
      "name": "798eb1c5-8054-497c-b466-6bdb523be072",
      "status": "Failed",
      "startTime": "2026-10-02T23:16:07.311984652Z",
      "endTime": "2026-10-02T23:35:13.399526187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bc2640>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bc2640>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xc0008d1b20>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-b86qkn" in resourcegroup="rg-np-version-upgrade-b86qkn-wdjflpkjvp8p" to finish creating: GET https://rp.j0034432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24",
      "name": "ba8abfad-4308-4775-a81e-7fccc827da24",
      "status": "Failed",
      "startTime": "2026-10-02T21:59:34.896590709Z",
      "endTime": "2026-10-02T22:18:38.199181809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xc0008d1b20>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-b86qkn" in resourcegroup="rg-np-version-upgrade-b86qkn-wdjflpkjvp8p" to finish creating: GET https://rp.j0034432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24",
      "name": "ba8abfad-4308-4775-a81e-7fccc827da24",
      "status": "Failed",
      "startTime": "2026-10-02T21:59:34.896590709Z",
      "endTime": "2026-10-02T22:18:38.199181809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.006s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-d97ndt" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc001ec6360>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.006s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-d97ndt" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc001ec6360>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001047368>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001047368>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T22:50:00Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0034432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0034432/providers/microsoft.monitor/accounts/services-j0034432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-fdbccb774-c7nq9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0034432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T22:50:00Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0034432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0034432/providers/microsoft.monitor/accounts/services-j0034432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-fdbccb774-c7nq9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0034432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc00022e5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6tx6ckwrt7wg" to finish creating: GET https://rp.j8116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113",
      "name": "2b9f2f0a-a724-431a-8648-3c5581a50113",
      "status": "Failed",
      "startTime": "2026-10-02T21:27:28.841238262Z",
      "endTime": "2026-10-02T21:46:29.997272093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc00022e5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-6tx6ckwrt7wg" to finish creating: GET https://rp.j8116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9f2f0a-a724-431a-8648-3c5581a50113",
      "name": "2b9f2f0a-a724-431a-8648-3c5581a50113",
      "status": "Failed",
      "startTime": "2026-10-02T21:27:28.841238262Z",
      "endTime": "2026-10-02T21:46:29.997272093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6tx6ckwrt7wg--managed/providers/Microsoft.Compute/virtualMachines/v3c9l1u3o0d3l2n-np-1-bhwvt-x7hn7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T21:34:19Z
  Ended:       2026-10-02T21:40:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8116608-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00378d080>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-46hf4d98wdxd" to finish creating: GET https://rp.j6618240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "name": "19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "status": "Failed",
      "startTime": "2026-10-02T21:11:48.464360183Z",
      "endTime": "2026-10-02T21:30:51.922878069Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00378d080>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-46hf4d98wdxd" to finish creating: GET https://rp.j6618240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "name": "19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "status": "Failed",
      "startTime": "2026-10-02T21:11:48.464360183Z",
      "endTime": "2026-10-02T21:30:51.922878069Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T20:53:37Z
  Ended:       2026-10-02T20:59:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6618240-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs"
Unexpected error:
    <*fmt.wrapError | 0xc00067a620>: 
    failed to create NodePool np-q47mgs: failed waiting for nodepool="np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs" in resourcegroup="rg-candidate-4-21-8nd2gr7hp9jn" to finish creating: GET https://rp.j6902144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d",
      "name": "fc132130-8033-44d4-826f-921357d1ad9d",
      "status": "Failed",
      "startTime": "2026-10-02T20:44:13.243887943Z",
      "endTime": "2026-10-02T21:03:17.125285435Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs"
Unexpected error:
    <*fmt.wrapError | 0xc00067a620>: 
    failed to create NodePool np-q47mgs: failed waiting for nodepool="np-q47mgs" for cluster "cluster-candidate-4-21-q47mgs" in resourcegroup="rg-candidate-4-21-8nd2gr7hp9jn" to finish creating: GET https://rp.j6902144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc132130-8033-44d4-826f-921357d1ad9d",
      "name": "fc132130-8033-44d4-826f-921357d1ad9d",
      "status": "Failed",
      "startTime": "2026-10-02T20:44:13.243887943Z",
      "endTime": "2026-10-02T21:03:17.125285435Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-8nd2gr7hp9jn-candidate-q47mgs--managed/providers/Microsoft.Compute/virtualMachines/q3s1n7f1d8j4z7u-np-q47mgs-sg5mc-vdkh2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001e820c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ftx74z7plhzf" to finish creating: GET https://rp.j1487616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "name": "737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "status": "Failed",
      "startTime": "2026-10-02T19:53:28.478249577Z",
      "endTime": "2026-10-02T20:12:35.176603143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001e820c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ftx74z7plhzf" to finish creating: GET https://rp.j1487616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "name": "737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "status": "Failed",
      "startTime": "2026-10-02T19:53:28.478249577Z",
      "endTime": "2026-10-02T20:12:35.176603143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002838820>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002838820>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 40 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T20:04:20Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-rqmvp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-rqmvp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T20:04:20Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-pclwm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-pclwm has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-ffmdk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-ffmdk has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-lqbwl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-lqbwl has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7xqdb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7xqdb has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-xfn2j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-xfn2j has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-hrb67", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-hrb67 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-9rzfn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-9rzfn has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-8rlch", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-8rlch has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-xz6lm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-xz6lm has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-psr9c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-psr9c has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-tdg92", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-tdg92 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nnzlj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nnzlj has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-cp4cg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-cp4cg has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-zs2jk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-zs2jk has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-bft9l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-bft9l has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-l6fg8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-l6fg8 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nkfbd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nkfbd has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-zqdt7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-zqdt7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-6hf78", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-6hf78 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-v2424", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-v2424 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7xlbs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7xlbs has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-gr2nd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-gr2nd has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-hhkrx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-hhkrx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-wnr5j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-wnr5j has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-6kxrp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-6kxrp has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-wxmwp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-wxmwp has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-dgvs2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-dgvs2 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nt5f5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nt5f5 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-qldfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-qldfs has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-vjmrq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-vjmrq has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-mz8lv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-mz8lv has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7r5rf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7r5rf has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-4ljs8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-4ljs8 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-vp8px", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-vp8px has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-t8hh6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-t8hh6 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-9xv2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-9xv2x has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-w4rxn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-w4rxn has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-mfxhq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-mfxhq has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-vngdx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-vngdx has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-c74lpp"
Unexpected error:
    <*fmt.wrapError | 0xc001776360>: 
    DNS for route host agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-c74lpp"
Unexpected error:
    <*fmt.wrapError | 0xc001776360>: 
    DNS for route host agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc00099b300>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8jgnh62lstlc" to finish creating: GET https://rp.j0313216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42",
      "name": "baa8a5de-ef44-437c-8139-ba1877f5af42",
      "status": "Failed",
      "startTime": "2026-10-02T19:34:49.936949596Z",
      "endTime": "2026-10-02T19:53:54.748504782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc00099b300>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8jgnh62lstlc" to finish creating: GET https://rp.j0313216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/baa8a5de-ef44-437c-8139-ba1877f5af42",
      "name": "baa8a5de-ef44-437c-8139-ba1877f5af42",
      "status": "Failed",
      "startTime": "2026-10-02T19:34:49.936949596Z",
      "endTime": "2026-10-02T19:53:54.748504782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8jgnh62lstlc--managed/providers/Microsoft.Compute/virtualMachines/a8h7l0q0o6h7k0x-np-1-l7qq4-bqbqw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T19:11:45Z
  Ended:       2026-10-02T19:19:41Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3162112-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000c38480>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-54tj44" in resourcegroup="rg-np-skip-minor-54tj44-rqt5qjd65czw" to finish creating: GET https://rp.j7680896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043",
      "name": "6d49abb2-b87c-4df6-b87c-df15e54da043",
      "status": "Failed",
      "startTime": "2026-10-02T19:28:01.557139687Z",
      "endTime": "2026-10-02T19:47:08.946643419Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000c38480>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-54tj44" in resourcegroup="rg-np-skip-minor-54tj44-rqt5qjd65czw" to finish creating: GET https://rp.j7680896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d49abb2-b87c-4df6-b87c-df15e54da043",
      "name": "6d49abb2-b87c-4df6-b87c-df15e54da043",
      "status": "Failed",
      "startTime": "2026-10-02T19:28:01.557139687Z",
      "endTime": "2026-10-02T19:47:08.946643419Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-54tj44-rqt5qjd65czw-np-sm-54tj44--managed/providers/Microsoft.Compute/virtualMachines/e5k1q1b6x4c4e8b-nps-420-422-g7bhw-kc47b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T19:07:37Z
  Ended:       2026-10-02T19:15:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7680896-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T18:37:05Z
  Ended:       2026-10-02T18:43:05Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5126912-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:55:55Z
  Ended:       2026-10-02T18:01:55Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4610176-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T19:11:26Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4610176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4610176/providers/microsoft.monitor/accounts/services-j4610176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-69f9b555fd-7q6cb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4610176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T19:11:26Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4610176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4610176/providers/microsoft.monitor/accounts/services-j4610176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-69f9b555fd-7q6cb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4610176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7b9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-92jbnctz6g9n" to finish creating: GET https://rp.j2828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "name": "16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "status": "Failed",
      "startTime": "2026-10-02T17:47:10.869193334Z",
      "endTime": "2026-10-02T18:06:19.043011596Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7b9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-92jbnctz6g9n" to finish creating: GET https://rp.j2828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "name": "16cd1ed0-a68c-40a4-ac3b-14d23a71ccf8",
      "status": "Failed",
      "startTime": "2026-10-02T17:47:10.869193334Z",
      "endTime": "2026-10-02T18:06:19.043011596Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-92jbnctz6g9n--managed/providers/Microsoft.Compute/virtualMachines/e0f5y6t9v5u2s0y-np-1-5bmkp-xqtnd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 65 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-22dl9", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-22dl9 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dfssq", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dfssq has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-lc7f8", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-lc7f8 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-lrxrn", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-lrxrn has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ws459", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ws459 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-nm4b8", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-nm4b8 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-r88sm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-r88sm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-2f87w", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-2f87w has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-hlcf4", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-hlcf4 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6mnrg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6mnrg has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-t95zp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-t95zp has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jkbjb", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jkbjb has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zcpgg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zcpgg has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-qxvct", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-qxvct has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-mfbkm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-mfbkm has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-mmpwv", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-mmpwv has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-m4v82", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-m4v82 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-fzf98", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-fzf98 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6f2b7", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6f2b7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-z9cp6", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-z9cp6 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-64mk2", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-64mk2 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-glxpz", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-glxpz has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-scxrj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-scxrj has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5876w", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5876w has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-h5skh", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-h5skh has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-77rvv", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-77rvv has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-wxzgc", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-wxzgc has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gxvrp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gxvrp has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-f497d", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-f497d has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zpfft", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zpfft has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-x99lx", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-x99lx has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ksr8r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ksr8r has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-p2k9c", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-p2k9c has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-hp5fb", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-hp5fb has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vjh72", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vjh72 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gfrgk", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gfrgk has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-nrh6c", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-nrh6c has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dcz7s", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dcz7s has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-56n2t", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-56n2t has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-8pwww", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-8pwww has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-sr95r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-sr95r has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-sl2gh", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-sl2gh has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dlbhp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dlbhp has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-pbjdw", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-pbjdw has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-rfjgp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-rfjgp has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-xwfwj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-xwfwj has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gl2xg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gl2xg has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-96lg5", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-96lg5 has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vbcxj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vbcxj has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5sw59", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5sw59 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-l49jm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-l49jm has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jx5v4", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jx5v4 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-s256n", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-s256n has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vnr48", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vnr48 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zrk5n", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zrk5n has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-cc7rs", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-cc7rs has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6j84r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6j84r has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-blmqs", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-blmqs has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-4lnm2", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-4lnm2 has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ppjmp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ppjmp has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-t9nwg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-t9nwg has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-68bhc", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-68bhc has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5fhnt", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5fhnt has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jjrn5", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jjrn5 has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-rzbk6", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-rzbk6 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:55:13Z
  Ended:       2026-10-02T18:02:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2828160-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011448e8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011448e8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.001s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-qjs7pm" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0011ce348>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.001s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-qjs7pm" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0011ce348>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:21:19Z
  Ended:       2026-10-02T17:32:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6990720-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6pa7oq39hmo04tlm81048rosc9slps-ea-listt98lhd", pod="kube-apiserver-594c7bf48-44xbg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6pa7oq39hmo04tlm81048rosc9slps-ea-listt98lhd/kube-apiserver-594c7bf48-44xbg has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:02:52Z
  Ended:       2026-10-02T17:08:49Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7516032-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000ca0a60>: 
    failed to create HCP cluster ea-listpkp2p7: failed waiting for cluster="ea-listpkp2p7" in resourcegroup="ea-list-4ccm8pbgrf47" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "name": "ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "status": "Failed",
      "startTime": "2026-10-02T15:58:05.107845034Z",
      "endTime": "2026-10-02T16:27:08.62997793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 1m14s ago; version 4.21.36 is Partial (want Completed), started 5m36s ago; version 4.21.36 is Partial (want Completed), started 11m4s ago; version 4.21.36 is Partial (want Completed), started 15m6s ago; version 4.21.36 is Partial (want Completed), started 20m46s ago; version 4.21.36 is Partial (want Completed), started 24m30s ago; version 4.21.36 is Partial (want Completed), started 25m39s ago; version 4.21.36 is Partial (want Completed), started 25m51s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000ca0a60>: 
    failed to create HCP cluster ea-listpkp2p7: failed waiting for cluster="ea-listpkp2p7" in resourcegroup="ea-list-4ccm8pbgrf47" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "name": "ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "status": "Failed",
      "startTime": "2026-10-02T15:58:05.107845034Z",
      "endTime": "2026-10-02T16:27:08.62997793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 1m14s ago; version 4.21.36 is Partial (want Completed), started 5m36s ago; version 4.21.36 is Partial (want Completed), started 11m4s ago; version 4.21.36 is Partial (want Completed), started 15m6s ago; version 4.21.36 is Partial (want Completed), started 20m46s ago; version 4.21.36 is Partial (want Completed), started 24m30s ago; version 4.21.36 is Partial (want Completed), started 25m39s ago; version 4.21.36 is Partial (want Completed), started 25m51s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014427a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-h248xrjjcrlg" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "name": "38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "status": "Failed",
      "startTime": "2026-10-02T16:37:00.371020776Z",
      "endTime": "2026-10-02T16:56:08.531132296Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014427a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-h248xrjjcrlg" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "name": "38b2c7b6-a5ca-4ed6-a22c-bfc22444845b",
      "status": "Failed",
      "startTime": "2026-10-02T16:37:00.371020776Z",
      "endTime": "2026-10-02T16:56:08.531132296Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-h248xrjjcrlg--managed/providers/Microsoft.Compute/virtualMachines/d6z7a4r5k4r5t0t-autoscale-noaz-8nr45-nxs6x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bdc160>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bdc160>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T16:13:37Z
  Ended:       2026-10-02T16:20:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4233600-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T16:10:23Z
  Ended:       2026-10-02T16:17:22Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7055872-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc0011632a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-sq4gl58phslg" to finish creating: GET https://rp.j7228672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "name": "1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "status": "Failed",
      "startTime": "2026-10-02T16:32:42.711328508Z",
      "endTime": "2026-10-02T16:51:42.25861049Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc0011632a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-sq4gl58phslg" to finish creating: GET https://rp.j7228672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "name": "1b3769b3-98f7-4259-8915-bd5c46b08c2e",
      "status": "Failed",
      "startTime": "2026-10-02T16:32:42.711328508Z",
      "endTime": "2026-10-02T16:51:42.25861049Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-sq4gl58phslg--managed/providers/Microsoft.Compute/virtualMachines/w8z1t1s1w4m9z0y-np-1-m2f5m-dvh7s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml"
Unexpected error:
    <*fmt.wrapError | 0xc000c8a7e0>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml" in resourcegroup="rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx" to finish creating: GET https://rp.j9111424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377",
      "name": "37ab050f-9ade-44b7-9470-1944f6bc1377",
      "status": "Failed",
      "startTime": "2026-10-02T16:08:46.628951449Z",
      "endTime": "2026-10-02T16:27:53.982628223Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml"
Unexpected error:
    <*fmt.wrapError | 0xc000c8a7e0>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-23-tjxdml" in resourcegroup="rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx" to finish creating: GET https://rp.j9111424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37ab050f-9ade-44b7-9470-1944f6bc1377",
      "name": "37ab050f-9ade-44b7-9470-1944f6bc1377",
      "status": "Failed",
      "startTime": "2026-10-02T16:08:46.628951449Z",
      "endTime": "2026-10-02T16:27:53.982628223Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-23-klzk5dvhl6cx-cp-ystream-de-fb92fcf5/providers/Microsoft.Compute/virtualMachines/z0i9p2v8n7r1u8z-np-deny-pods-hq6c9-vlq6k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T15:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-k7gkp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-k7gkp has been in a non-ready state for longer than 5 minutes.

alert fired 14 time(s) (2 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T16:07:06Z
  Ended:       2026-10-02T16:28:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6ocek7h2b81l310othj19g5cs2hvdm-h6f8s9c6d5a8t7c", pod="router-7466989cdd-4xgts", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6ocek7h2b81l310othj19g5cs2hvdm-h6f8s9c6d5a8t7c/router-7466989cdd-4xgts has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T16:07:07Z
  Ended:       2026-10-02T16:28:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6oclvhiuo0ben6245b5rc60j4ul3v1-v20261001", pod="router-58b59f844f-6fgw8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6oclvhiuo0ben6245b5rc60j4ul3v1-v20261001/router-58b59f844f-6fgw8 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs"
Unexpected error:
    <*fmt.wrapError | 0xc000bb3a80>: 
    failed to create NodePool np-xbxlzs: failed waiting for nodepool="np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs" in resourcegroup="rg-candidate-4-20-gwcqh9m2pngx" to finish creating: GET https://rp.j2125184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971",
      "name": "2e698a08-7d53-4eac-b66e-e0b997094971",
      "status": "Failed",
      "startTime": "2026-10-02T15:16:45.725296348Z",
      "endTime": "2026-10-02T15:35:49.304545041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs"
Unexpected error:
    <*fmt.wrapError | 0xc000bb3a80>: 
    failed to create NodePool np-xbxlzs: failed waiting for nodepool="np-xbxlzs" for cluster "cluster-candidate-4-20-xbxlzs" in resourcegroup="rg-candidate-4-20-gwcqh9m2pngx" to finish creating: GET https://rp.j2125184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e698a08-7d53-4eac-b66e-e0b997094971",
      "name": "2e698a08-7d53-4eac-b66e-e0b997094971",
      "status": "Failed",
      "startTime": "2026-10-02T15:16:45.725296348Z",
      "endTime": "2026-10-02T15:35:49.304545041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwcqh9m2pngx-candidate-xbxlzs--managed/providers/Microsoft.Compute/virtualMachines/w9p4z8q1b2g0x7t-np-xbxlzs-7kn5c-mb724 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T15:21:16Z
  Ended:       2026-10-02T15:28:15Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2125184-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T15:09:09Z
  Ended:       2026-10-02T15:15:04Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6816000-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T15:01:55Z
  Ended:       2026-10-02T15:07:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9080064-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T13:57:08Z
  Ended:       2026-10-02T14:15:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4702592-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00156e168>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00156e168>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000a14dc0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-4jgj2xjf9rbf" to finish creating: GET https://rp.j6116992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826",
      "name": "c833939d-30b9-4409-82fd-33d2b22e8826",
      "status": "Failed",
      "startTime": "2026-10-02T13:10:20.783755056Z",
      "endTime": "2026-10-02T13:29:28.266052933Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000a14dc0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-4jgj2xjf9rbf" to finish creating: GET https://rp.j6116992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c833939d-30b9-4409-82fd-33d2b22e8826",
      "name": "c833939d-30b9-4409-82fd-33d2b22e8826",
      "status": "Failed",
      "startTime": "2026-10-02T13:10:20.783755056Z",
      "endTime": "2026-10-02T13:29:28.266052933Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4jgj2xjf9rbf--managed/providers/Microsoft.Compute/virtualMachines/a7g2b1n4e0w3b0d-autoscale-noaz-xqpdr-gd9vj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000c23e80>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.004s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000c23e80>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.004s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0028843f0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kqrgzxwnxwgd" to finish creating: GET https://rp.j8271104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "name": "7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "status": "Failed",
      "startTime": "2026-10-02T12:59:29.315762309Z",
      "endTime": "2026-10-02T13:18:36.006994146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-597b9-prdsc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-597b9-prdsc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0028843f0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kqrgzxwnxwgd" to finish creating: GET https://rp.j8271104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "name": "7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "status": "Failed",
      "startTime": "2026-10-02T12:59:29.315762309Z",
      "endTime": "2026-10-02T13:18:36.006994146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-597b9-prdsc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-597b9-prdsc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:212]: Timed out after 2700.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:210 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:212]: Timed out after 2700.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:210 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gc94n755ktzv/cluster-candidate-5-0-l6z5kw should provision
Unexpected error:
    <*fmt.wrapError | 0xc001032d80>: 
    failed to create HCP cluster cluster-candidate-5-0-l6z5kw: failed waiting for cluster="cluster-candidate-5-0-l6z5kw" in resourcegroup="rg-candidate-5-0-gc94n755ktzv" to finish creating: GET https://rp.j1697920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "name": "af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "status": "Failed",
      "startTime": "2026-10-02T11:49:05.164042474Z",
      "endTime": "2026-10-02T12:18:11.434067357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gc94n755ktzv/cluster-candidate-5-0-l6z5kw should provision
Unexpected error:
    <*fmt.wrapError | 0xc001032d80>: 
    failed to create HCP cluster cluster-candidate-5-0-l6z5kw: failed waiting for cluster="cluster-candidate-5-0-l6z5kw" in resourcegroup="rg-candidate-5-0-gc94n755ktzv" to finish creating: GET https://rp.j1697920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "name": "af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "status": "Failed",
      "startTime": "2026-10-02T11:49:05.164042474Z",
      "endTime": "2026-10-02T12:18:11.434067357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:19:46Z
  Ended:       2026-10-02T12:37:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T12:19:46Z
  Ended:       2026-10-02T12:37:46Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:24:46Z
  Ended:       2026-10-02T12:39:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:11:46Z
  Ended:       2026-10-02T12:41:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3ec768d5-6cf8-4b9c-a960-342894bf7da7"
  Description: Pod ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-02T12:11:46Z
  Ended:       2026-10-02T12:41:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3ec768d5-6cf8-4b9c-a960-342894bf7da7"
  Description: Pod ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010e1940>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-w27q7sgvxjfv" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699",
      "name": "d3cd18ff-072e-464f-89ca-f4c27910d699",
      "status": "Failed",
      "startTime": "2026-10-02T11:50:03.491486549Z",
      "endTime": "2026-10-02T12:09:10.243715073Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010e1940>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-w27q7sgvxjfv" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3cd18ff-072e-464f-89ca-f4c27910d699",
      "name": "d3cd18ff-072e-464f-89ca-f4c27910d699",
      "status": "Failed",
      "startTime": "2026-10-02T11:50:03.491486549Z",
      "endTime": "2026-10-02T12:09:10.243715073Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-w27q7sgvxjfv--managed/providers/Microsoft.Compute/virtualMachines/s3a6t6z4u9a9y2v-nodepool-128gib-7vpll-jwmrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000f670c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-tjrqglfwz7jf" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "name": "92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "status": "Failed",
      "startTime": "2026-10-02T11:51:54.872760103Z",
      "endTime": "2026-10-02T12:11:00.580352219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine acr-pull-create-np-1-rg8t6-jcb2q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine acr-pull-create-np-1-rg8t6-jcb2q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:247]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0xc000f670c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull-create" in resourcegroup="acr-pull-create-tjrqglfwz7jf" to finish creating: GET https://rp.j0488832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "name": "92cbdbab-88df-42a5-95a1-670ebdd455b3",
      "status": "Failed",
      "startTime": "2026-10-02T11:51:54.872760103Z",
      "endTime": "2026-10-02T12:11:00.580352219Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine acr-pull-create-np-1-rg8t6-jcb2q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine acr-pull-create-np-1-rg8t6-jcb2q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-tjrqglfwz7jf--managed/providers/Microsoft.Compute/virtualMachines/acr-pull-create-np-1-rg8t6-jcb2q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T11:59:15Z
  Ended:       2026-10-02T12:14:18Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0488832-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002032960>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-s47ln9cbqhqp" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "name": "b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "status": "Failed",
      "startTime": "2026-10-02T11:09:22.074259734Z",
      "endTime": "2026-10-02T11:28:22.963531957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002032960>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-s47ln9cbqhqp" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "name": "b17a89af-b196-4b23-bd98-9d9ab5a5f12a",
      "status": "Failed",
      "startTime": "2026-10-02T11:09:22.074259734Z",
      "endTime": "2026-10-02T11:28:22.963531957Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-kcxkl-4xf9f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-s47ln9cbqhqp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-kcxkl-4xf9f to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.009s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-dg8wqw" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0015c3050>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.009s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-dg8wqw" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0015c3050>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000dc15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000dc15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00063e600>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-snc8bz4l2lpm" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999",
      "name": "377e91a7-9fb9-4364-a4c3-94d301101999",
      "status": "Failed",
      "startTime": "2026-10-02T11:41:21.075539741Z",
      "endTime": "2026-10-02T12:00:23.993693871Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00063e600>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-snc8bz4l2lpm" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999",
      "name": "377e91a7-9fb9-4364-a4c3-94d301101999",
      "status": "Failed",
      "startTime": "2026-10-02T11:41:21.075539741Z",
      "endTime": "2026-10-02T12:00:23.993693871Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:193]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015c66c0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nqllmtrxwh84" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-nqllmtrxwh84, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:193]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015c66c0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nqllmtrxwh84" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-nqllmtrxwh84, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:416]: failed to poll cluster "cidr-connectivity-test" authorized CIDRs update to completion (timeout '10.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:416]: failed to poll cluster "cidr-connectivity-test" authorized CIDRs update to completion (timeout '10.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9"
Unexpected error:
    <*fmt.wrapError | 0xc001238cc0>: 
    failed to create NodePool np-j4v4c9: failed waiting for nodepool="np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9" in resourcegroup="rg-candidate-4-20-ck8mc4szhlcn" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "name": "f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "status": "Failed",
      "startTime": "2026-10-02T11:08:30.228624897Z",
      "endTime": "2026-10-02T11:27:32.877879167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9"
Unexpected error:
    <*fmt.wrapError | 0xc001238cc0>: 
    failed to create NodePool np-j4v4c9: failed waiting for nodepool="np-j4v4c9" for cluster "cluster-candidate-4-20-j4v4c9" in resourcegroup="rg-candidate-4-20-ck8mc4szhlcn" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "name": "f0d415e4-e138-4042-a3b0-d8c32121b5e0",
      "status": "Failed",
      "startTime": "2026-10-02T11:08:30.228624897Z",
      "endTime": "2026-10-02T11:27:32.877879167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ck8mc4szhlcn-candidate-j4v4c9--managed/providers/Microsoft.Compute/virtualMachines/m4v6j6x4k0f0u5k-np-j4v4c9-h2pws-dlbbq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d8120>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-9h5ws9hlzq4p" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-9h5ws9hlzq4p, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d8120>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-9h5ws9hlzq4p" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-9h5ws9hlzq4p, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*fmt.wrapErrors | 0xc0013d63c0>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d4f6lxz4s2dz" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-d4f6lxz4s2dz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*fmt.wrapErrors | 0xc0013d63c0>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d4f6lxz4s2dz" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-d4f6lxz4s2dz, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:369]: failed to update containerRegistry via PATCH
Unexpected error:
    <*fmt.wrapErrors | 0xc000fa8690>: 
    failed waiting for hcpCluster="acr-pull-create" in resourcegroup="acr-pull-create-ljhpblch2468" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20261001 for cluster acr-pull-create in resource group acr-pull-create-ljhpblch2468, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:369]: failed to update containerRegistry via PATCH
Unexpected error:
    <*fmt.wrapErrors | 0xc000fa8690>: 
    failed waiting for hcpCluster="acr-pull-create" in resourcegroup="acr-pull-create-ljhpblch2468" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20261001 for cluster acr-pull-create in resource group acr-pull-create-ljhpblch2468, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T11:16:17Z
  Ended:       2026-10-02T11:26:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7658624-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6k02kauka61vjc7pvhhdhndnsmtknd-ea-list6nlmtx", pod="kube-apiserver-67bb546754-b454t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6k02kauka61vjc7pvhhdhndnsmtknd-ea-list6nlmtx/kube-apiserver-67bb546754-b454t has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T09:12:36Z
  Ended:       2026-10-02T09:27:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9667072-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T09:12:36Z
  Ended:       2026-10-02T09:30:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9667072-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-10-02T09:12:36Z
  Ended:       2026-10-02T09:28:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9667072-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000b84f00>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2vzgl45q9ll4" to finish creating: GET https://rp.j0616704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "name": "992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "status": "Failed",
      "startTime": "2026-10-02T08:27:53.917230714Z",
      "endTime": "2026-10-02T08:46:59.056787427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000b84f00>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2vzgl45q9ll4" to finish creating: GET https://rp.j0616704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "name": "992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "status": "Failed",
      "startTime": "2026-10-02T08:27:53.917230714Z",
      "endTime": "2026-10-02T08:46:59.056787427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T08:10:58Z
  Ended:       2026-10-02T08:29:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0616704-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh"
Unexpected error:
    <*fmt.wrapError | 0xc000eb30a0>: 
    failed to create NodePool np-7ctmvh: failed waiting for nodepool="np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh" in resourcegroup="rg-candidate-4-23-ztlck247gbc7" to finish creating: GET https://rp.j6550784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482",
      "name": "28de2827-3d1e-473f-9ac1-7443442e5482",
      "status": "Failed",
      "startTime": "2026-10-02T06:54:46.375852677Z",
      "endTime": "2026-10-02T07:13:49.710009064Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh"
Unexpected error:
    <*fmt.wrapError | 0xc000eb30a0>: 
    failed to create NodePool np-7ctmvh: failed waiting for nodepool="np-7ctmvh" for cluster "cluster-candidate-4-23-7ctmvh" in resourcegroup="rg-candidate-4-23-ztlck247gbc7" to finish creating: GET https://rp.j6550784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/28de2827-3d1e-473f-9ac1-7443442e5482",
      "name": "28de2827-3d1e-473f-9ac1-7443442e5482",
      "status": "Failed",
      "startTime": "2026-10-02T06:54:46.375852677Z",
      "endTime": "2026-10-02T07:13:49.710009064Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-ztlck247gbc7-candidate-7ctmvh--managed/providers/Microsoft.Compute/virtualMachines/r9l8y1w3n6j7l5a-np-7ctmvh-v8j29-2gnt5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-10-02T04:18:24.964Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-02T04:18:25.604Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:3a6b5dce-c503-483f-868b-3656736645ed cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j6706176.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j6706176-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j6706176-1 secret:hcp-underlay-j6706176-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-02T04:18:25.604Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:18:26.032Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-02T04:18:26.347Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-02T04:18:26.347Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:18:26.470Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:23:32.794Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:23:32.795Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000e9d1e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-z847fd" in resourcegroup="rg-np-version-upgrade-z847fd-9d7c47wqwcb4" to finish creating: GET https://rp.j1887616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "name": "6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "status": "Failed",
      "startTime": "2026-10-02T04:46:32.427711572Z",
      "endTime": "2026-10-02T05:05:35.48212253Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000e9d1e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-z847fd" in resourcegroup="rg-np-version-upgrade-z847fd-9d7c47wqwcb4" to finish creating: GET https://rp.j1887616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "name": "6b15e942-32f6-4100-897e-aeb4bb2b3983",
      "status": "Failed",
      "startTime": "2026-10-02T04:46:32.427711572Z",
      "endTime": "2026-10-02T05:05:35.48212253Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z847fd-9d7c47wqwcb4-np-upgrade-z8-2e598309/providers/Microsoft.Compute/virtualMachines/q0m2l9l1f2b4p9g-npupgrade-4-20-5rnx4-mqdsr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-10-02T04:04:12.931Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-02T04:04:13.290Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:ad930020-a0d3-4a84-b682-06665523a404 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j3304448.canadacentral-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j3304448-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j3304448-1 secret:hcp-underlay-j3304448-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-02T04:04:13.290Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:04:13.437Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-02T04:04:13.594Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-02T04:04:13.594Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:04:13.638Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:09:17.935Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:09:17.936Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-wzq9v", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-wzq9v has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-qj4ln", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-qj4ln has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-w44fw", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-w44fw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-66rf9", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-66rf9 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-4p6tc", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-4p6tc has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000f90e60>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-gxq8mc" in resourcegroup="rg-np-skip-minor-gxq8mc-kwv67gxfnh4r" to finish creating: GET https://rp.j3775744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "name": "5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "status": "Failed",
      "startTime": "2026-10-02T04:52:30.925064823Z",
      "endTime": "2026-10-02T05:11:35.851064932Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xc000f90e60>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-gxq8mc" in resourcegroup="rg-np-skip-minor-gxq8mc-kwv67gxfnh4r" to finish creating: GET https://rp.j3775744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "name": "5dd38511-4f3f-4db5-ba8c-93507fe57cde",
      "status": "Failed",
      "startTime": "2026-10-02T04:52:30.925064823Z",
      "endTime": "2026-10-02T05:11:35.851064932Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-gxq8mc-kwv67gxfnh4r-np-sm-gxq8mc--managed/providers/Microsoft.Compute/virtualMachines/v0b4f8d8t1c5i8g-nps-420-422-7mmqj-6nqb7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000fef1c0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-svbr9q" in resourcegroup="rg-np-downgrade-svbr9q-kvqmb7zbtpsq" to finish creating: GET https://rp.j3810048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "name": "556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "status": "Failed",
      "startTime": "2026-10-02T01:38:44.187977267Z",
      "endTime": "2026-10-02T01:57:50.438850256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000fef1c0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-svbr9q" in resourcegroup="rg-np-downgrade-svbr9q-kvqmb7zbtpsq" to finish creating: GET https://rp.j3810048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "name": "556c3a8a-eee8-47cd-95b3-aed05e7e4d22",
      "status": "Failed",
      "startTime": "2026-10-02T01:38:44.187977267Z",
      "endTime": "2026-10-02T01:57:50.438850256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-svbr9q-kvqmb7zbtpsq-np-dg-svbr9q--managed/providers/Microsoft.Compute/virtualMachines/j0k8m5q0v6j7v3w-npdg-4-21-9gbl6-8hn7c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred