fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0x2f72d213c320>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-n6bsrg" in resourcegroup="rg-np-noedge-n6bsrg-vxj5p9qrl42z" to finish creating: GET https://rp.j2350080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b",
      "name": "83d90b9f-3285-4912-9884-40a78690862b",
      "status": "Failed",
      "startTime": "2026-10-04T00:37:10.755457782Z",
      "endTime": "2026-10-04T00:56:17.299103912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0x2f72d213c320>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-n6bsrg" in resourcegroup="rg-np-noedge-n6bsrg-vxj5p9qrl42z" to finish creating: GET https://rp.j2350080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b",
      "name": "83d90b9f-3285-4912-9884-40a78690862b",
      "status": "Failed",
      "startTime": "2026-10-04T00:37:10.755457782Z",
      "endTime": "2026-10-04T00:56:17.299103912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T22:09:35Z
  Ended:       2026-10-03T22:15:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0690944-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T21:07:34Z
  Ended:       2026-10-03T21:13:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1818624-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3528a4bdeac0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vljzr5rw54g9" to finish creating: GET https://rp.j7922048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e",
      "name": "336a4045-fdd2-4a2c-b138-8d99b939169e",
      "status": "Failed",
      "startTime": "2026-10-03T20:19:49.133571808Z",
      "endTime": "2026-10-03T20:38:58.66181942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 3 of 3 machines are not healthy\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 3 of 3 machines are not ready\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3528a4bdeac0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vljzr5rw54g9" to finish creating: GET https://rp.j7922048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e",
      "name": "336a4045-fdd2-4a2c-b138-8d99b939169e",
      "status": "Failed",
      "startTime": "2026-10-03T20:19:49.133571808Z",
      "endTime": "2026-10-03T20:38:58.66181942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 3 of 3 machines are not healthy\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 3 of 3 machines are not ready\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T19:53:25Z
  Ended:       2026-10-03T19:59:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7836160-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1590b4a34e00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-hd5zgd7fw6ql" to finish creating: GET https://rp.j0239360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94",
      "name": "0504a146-7d92-49ac-9646-6e1b202d7d94",
      "status": "Failed",
      "startTime": "2026-10-03T18:22:53.322060806Z",
      "endTime": "2026-10-03T18:41:55.898243278Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkgjk-bhjgr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkgjk-bhjgr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1590b4a34e00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-hd5zgd7fw6ql" to finish creating: GET https://rp.j0239360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94",
      "name": "0504a146-7d92-49ac-9646-6e1b202d7d94",
      "status": "Failed",
      "startTime": "2026-10-03T18:22:53.322060806Z",
      "endTime": "2026-10-03T18:41:55.898243278Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkgjk-bhjgr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkgjk-bhjgr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T18:23:19Z
  Ended:       2026-10-03T18:38:18Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0143360-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-file-csi-driver-operator-56bd98ff47-n9mw7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-file-csi-driver-operator-56bd98ff47-n9mw7 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-disk-csi-driver-operator-68446dbd4d-gp5r7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-disk-csi-driver-operator-68446dbd4d-gp5r7 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-disk-csi-driver-operator-68446dbd4d-72x46", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-disk-csi-driver-operator-68446dbd4d-72x46 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-disk-csi-driver-operator-68446dbd4d-l8w9v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-disk-csi-driver-operator-68446dbd4d-l8w9v has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-file-csi-driver-operator-56bd98ff47-lhdbd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-file-csi-driver-operator-56bd98ff47-lhdbd has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xd9700701640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-l9xkxj2bpszs" to finish creating: GET https://rp.j1091968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "name": "d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "status": "Failed",
      "startTime": "2026-10-03T17:56:20.630751478Z",
      "endTime": "2026-10-03T18:15:20.975618436Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xd9700701640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-l9xkxj2bpszs" to finish creating: GET https://rp.j1091968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "name": "d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "status": "Failed",
      "startTime": "2026-10-03T17:56:20.630751478Z",
      "endTime": "2026-10-03T18:15:20.975618436Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 69 time(s) (63 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-52l2j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-52l2j has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-vdz8t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-vdz8t has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-k2787", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-k2787 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-zp9nc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-zp9nc has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-vk8sg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-vk8sg has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-fvs76", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-fvs76 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-llstd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-llstd has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-5mcng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-5mcng has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9sqdg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9sqdg has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:51:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-btfk4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-btfk4 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-dq2vz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-dq2vz has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-x6wz7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-x6wz7 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-ctpt9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-ctpt9 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-j2d9z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-j2d9z has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-4h79j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-4h79j has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-z2w4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-z2w4b has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-nkt4w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-nkt4w has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-ffhbr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-ffhbr has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-csvsv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-csvsv has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-s4h6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-s4h6v has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-dl5l4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-dl5l4 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-47862", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-47862 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-gw8rm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-gw8rm has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-mrcs5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-mrcs5 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-tzs9x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-tzs9x has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-pjk6c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-pjk6c has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-vdtmc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-vdtmc has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-7kmn4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-7kmn4 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-6w4zc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-6w4zc has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-mnfxd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-mnfxd has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-m84qg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-m84qg has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-j8kmn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-j8kmn has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-r827x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-r827x has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lscgk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lscgk has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-gj4v6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-gj4v6 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-2mvqm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-2mvqm has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lw6l6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lw6l6 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9w425", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9w425 has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-ndzg5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-ndzg5 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-b662v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-b662v has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-wj72x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-wj72x has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-5c5cd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-5c5cd has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lkvbz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lkvbz has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-22klt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-22klt has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-rxm4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-rxm4b has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-r7qwh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-r7qwh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-xhl5w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-xhl5w has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-h4p6b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-h4p6b has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-j7f54", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-j7f54 has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-bs5j7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-bs5j7 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lwcng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lwcng has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9n44n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9n44n has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-dp4wn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-dp4wn has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-8qlz4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-8qlz4 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9n2td", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9n2td has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-d6x6h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-d6x6h has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-92ktd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-92ktd has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-sxxzw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-sxxzw has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-zvv2v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-zvv2v has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-2hsfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-2hsfs has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-nz8k8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-nz8k8 has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-gssl7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-gssl7 has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-7865d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-7865d has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x1737a1124260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kjgdvvpr4g6r" to finish creating: GET https://rp.j0472448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a",
      "name": "9760730d-edb7-41c2-8626-62f85b63fc1a",
      "status": "Failed",
      "startTime": "2026-10-03T16:34:46.276618892Z",
      "endTime": "2026-10-03T16:53:53.086174722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x1737a1124260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kjgdvvpr4g6r" to finish creating: GET https://rp.j0472448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a",
      "name": "9760730d-edb7-41c2-8626-62f85b63fc1a",
      "status": "Failed",
      "startTime": "2026-10-03T16:34:46.276618892Z",
      "endTime": "2026-10-03T16:53:53.086174722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x22d6ad451998>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x22d6ad451998>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T16:03:43Z
  Ended:       2026-10-03T16:09:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9578752-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-vzb878crbjw5" to finish creating: GET https://rp.j0614912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "name": "737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "status": "Failed",
  "startTime": "2026-10-03T14:04:39.339033685Z",
  "endTime": "2026-10-03T14:23:40.256240312Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-vzb878crbjw5" to finish creating: GET https://rp.j0614912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "name": "737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "status": "Failed",
  "startTime": "2026-10-03T14:04:39.339033685Z",
  "endTime": "2026-10-03T14:23:40.256240312Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x1d83c5b77a40>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-tzccwkmbcld2" to finish creating: GET https://rp.j4137216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70",
      "name": "63335228-135d-4644-b7d4-651deba91b70",
      "status": "Failed",
      "startTime": "2026-10-03T13:14:48.144915783Z",
      "endTime": "2026-10-03T13:33:57.141188553Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-47kjt-tgj5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-47kjt-tgj5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x1d83c5b77a40>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-tzccwkmbcld2" to finish creating: GET https://rp.j4137216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70",
      "name": "63335228-135d-4644-b7d4-651deba91b70",
      "status": "Failed",
      "startTime": "2026-10-03T13:14:48.144915783Z",
      "endTime": "2026-10-03T13:33:57.141188553Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-47kjt-tgj5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-47kjt-tgj5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-10-03T12:18:04.762Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 description="Step swift-vnet\n  Kind: Shell\n  Command: ./swift-vnet.sh\n"
time=2026-10-03T12:18:05.175Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: Failed to load or parse file /tmp/azure-cli-config-3770628587/azureProfile.json. It will be overridden by default settings.\nERROR: The subscription of 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' doesn't exist in cloud 'AzureCloud'.\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2d36ccbc2160>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vgn28b5mgscc" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "name": "1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "status": "Failed",
      "startTime": "2026-10-03T13:01:15.160971572Z",
      "endTime": "2026-10-03T13:20:17.766960003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2d36ccbc2160>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vgn28b5mgscc" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "name": "1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "status": "Failed",
      "startTime": "2026-10-03T13:01:15.160971572Z",
      "endTime": "2026-10-03T13:20:17.766960003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x227e9786b20>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-9q2trs" in resourcegroup="rg-np-skip-minor-9q2trs-89955s58px2l" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "name": "8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "status": "Failed",
      "startTime": "2026-10-03T12:59:28.477851902Z",
      "endTime": "2026-10-03T13:18:37.714240199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x227e9786b20>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-9q2trs" in resourcegroup="rg-np-skip-minor-9q2trs-89955s58px2l" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "name": "8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "status": "Failed",
      "startTime": "2026-10-03T12:59:28.477851902Z",
      "endTime": "2026-10-03T13:18:37.714240199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T12:40:19Z
  Ended:       2026-10-03T13:01:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5049088-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T10:35:51Z
  Ended:       2026-10-03T10:41:49Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4618112-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1ccede026220>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-74v4krpx78dr" to finish creating: GET https://rp.j7088384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551",
      "name": "61d03357-4fb7-49b2-a777-a99ffa662551",
      "status": "Failed",
      "startTime": "2026-10-03T10:52:21.517435656Z",
      "endTime": "2026-10-03T11:11:28.16247644Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1ccede026220>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-74v4krpx78dr" to finish creating: GET https://rp.j7088384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551",
      "name": "61d03357-4fb7-49b2-a777-a99ffa662551",
      "status": "Failed",
      "startTime": "2026-10-03T10:52:21.517435656Z",
      "endTime": "2026-10-03T11:11:28.16247644Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T09:10:27Z
  Ended:       2026-10-03T09:28:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1175680-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T09:15:27Z
  Ended:       2026-10-03T09:28:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1175680-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T08:23:18Z
  Ended:       2026-10-03T08:30:15Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7356672-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0x1229c1294580>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-q4mxjczq7tsd" to finish creating: GET https://rp.j2783744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080",
      "name": "7757c823-88ba-458e-aed3-9dfdac2d7080",
      "status": "Failed",
      "startTime": "2026-10-03T05:03:30.952777318Z",
      "endTime": "2026-10-03T05:22:34.856876402Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0x1229c1294580>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-q4mxjczq7tsd" to finish creating: GET https://rp.j2783744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080",
      "name": "7757c823-88ba-458e-aed3-9dfdac2d7080",
      "status": "Failed",
      "startTime": "2026-10-03T05:03:30.952777318Z",
      "endTime": "2026-10-03T05:22:34.856876402Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T05:06:36Z
  Ended:       2026-10-03T05:12:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2783744-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x331a5ee39be0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-hzfdbcnlckj2" to finish creating: GET https://rp.j8366592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "name": "f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "status": "Failed",
      "startTime": "2026-10-03T05:21:56.267331547Z",
      "endTime": "2026-10-03T05:41:00.127542846Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x331a5ee39be0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-hzfdbcnlckj2" to finish creating: GET https://rp.j8366592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "name": "f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "status": "Failed",
      "startTime": "2026-10-03T05:21:56.267331547Z",
      "endTime": "2026-10-03T05:41:00.127542846Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T04:38:45Z
  Ended:       2026-10-03T04:44:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3883264-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-03T04:27:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j3883264-svc", component="backend", name="openshiftversioncatalog", region="westus3", severity="warning"
  Description: Backend controller workqueue openshiftversioncatalog has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0x1113bcb9c160>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-rckbfw" in resourcegroup="rg-np-downgrade-rckbfw-856zsqvm5rkb" to finish creating: GET https://rp.j7360256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4",
      "name": "074aacd3-7322-43b0-9305-f06220347ff4",
      "status": "Failed",
      "startTime": "2026-10-03T04:27:23.785116321Z",
      "endTime": "2026-10-03T04:46:25.26888513Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0x1113bcb9c160>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-rckbfw" in resourcegroup="rg-np-downgrade-rckbfw-856zsqvm5rkb" to finish creating: GET https://rp.j7360256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4",
      "name": "074aacd3-7322-43b0-9305-f06220347ff4",
      "status": "Failed",
      "startTime": "2026-10-03T04:27:23.785116321Z",
      "endTime": "2026-10-03T04:46:25.26888513Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T04:07:22Z
  Ended:       2026-10-03T04:13:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7360256-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T03:07:49Z
  Ended:       2026-10-03T03:19:50Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1034112-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T03:08:45Z
  Ended:       2026-10-03T03:22:45Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1034112-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3263781c03a0>: 
    DNS for route host agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3263781c03a0>: 
    DNS for route host agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T01:56:32Z
  Ended:       2026-10-03T02:03:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4629376-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T01:57:33Z
  Ended:       2026-10-03T02:10:32Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4629376-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-xt2c7s"
Unexpected error:
    <*fmt.wrapError | 0x1b2c75ad8a80>: 
    DNS for route host agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-xt2c7s"
Unexpected error:
    <*fmt.wrapError | 0x1b2c75ad8a80>: 
    DNS for route host agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T02:01:45Z
  Ended:       2026-10-03T02:07:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6512256-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-10-03T01:35:28.208Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-10-03T01:35:28.778Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy values="map[azureArmHelperIdentityCertName:armHelperPoolCert-10 azureArmHelperIdentityClientId:0a9e3940-6bc3-4cbf-a8f7-c65c3d8d2462 azureArmHelperMockFpaPrincipalId:47f69502-0065-4d9a-b19b-d403e183d2f4 azureCloudEnvironmentName:AzurePublicCloud azureCsMiClientId:de4f7be2-47e3-4002-adc2-77c2fcdec830 azureFirstPartyApplicationClientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX azureMiMockServicePrincipalCertName:msiMockPoolCert-17 azureMiMockServicePrincipalClientId:e7112f7f-5b45-480e-bc44-abf9f219e6f0 azureMiMockServicePrincipalPrincipalId:39eefc4f-5e24-491d-9f09-bcc0ad573bcf azureOperatorsMI:map[dev:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] public:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] roleSetName:dev] backplaneURL: batchProcesses: batchProcessesDryRun:true clientId:foo clientScopes:openid clientSecret:bar clusterErrorReport:false cosmosDatabaseName:arohcpci01-rp-j4407168 cosmosDatabaseUrl:https://arohcpci01-rp-j4407168.documents.azure.com:443/ cpuRequest:630m csDeploymentStrategy:map[rollingUpdate:map[maxSurge:1 maxUnavailable:0] type:RollingUpdate] dataPlaneHAProxyImage:arohcpocpdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 databaseAuthMethod:postgres databaseDisableTls:true databaseHost:ocm-cs-db databaseK8sSecretName:ocm-cs-db databaseName:ocm-cs-db databasePassword:TheBlurstOfTimes databasePort:5432 databaseServiceName:ocm-cs-db databaseUser:ocm databaseVolumeCapacity:512Mi debugMaxDuration:5m debugPort: defaultExpiration:0 deployment:map[zoneCount:3] environment:arohcpci01 forceMigration: fpaCertName:firstPartyCert2 gatewayUrl:http://127.0.0.1:9090 global:map[imageDigest:sha256:5fffce295f8b49be38ae2385714d09af29700dd3dc60c40a058f7aeb66e97d72] imageDigest:sha256:5fffce295f8b49be38ae2385714d09af29700dd3dc60c40a058f7aeb66e97d72 imageRegistry:arohcpsvcdev.azurecr.io imageRepository:app-sre/aro-hcp-clusters-service insecure:false jwksUrl:http://localhost logLevel:4 machinePoolMigrationWorkerPeriod:1h managedIdentitiesDataPlaneAudienceResource:https://dummy.org maximumExpiration:0 memoryLimit:unlimited memoryRequest:150Mi ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev ocpAcrResourceUrl:arohcpocpdev.azurecr.io oidcIssuerBaseUrl:https://arohcpci01oidcj4407168.z9.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj4407168.blob.core.windows.net/ postgresqlVersion:12 provisionShardClusterLimit:100 pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:app-sre/aro-hcp-clusters-service:pull workloadIdentityClientId:27f533be-4c23-4032-9d19-da9c7454bd3b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] region:canadacentral replicas:3 runtimeMode:aro-hcp serviceAccountName:clusters-service serviceKeyVaultName:aro-hcp-dev-svc-kv tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesEnabled:true tlsCertificatesIssuer:Self tokenUrl:http://localhost useAzureDB:false userDefinedDnsBaseDomain:i1.devshift.org]"
time=2026-10-03T01:35:28.779Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-03T01:35:29.089Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy namespace=clusters-service
time=2026-10-03T01:35:29.415Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy dryRun=false
time=2026-10-03T01:35:29.416Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-03T01:35:29.465Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-03T01:45:35.236Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-10-03T01:45:35.237Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-03T01:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4407168-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-74695d59f9-hxdpn", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-74695d59f9-hxdpn has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-03T01:42:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4407168-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-74695d59f9-bfnvs", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-74695d59f9-bfnvs has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-03T01:42:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4407168-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-74695d59f9-wjv8w", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-74695d59f9-wjv8w has been in a non-ready state for longer than 5 minutes.