fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0x2f72d213c320>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-n6bsrg" in resourcegroup="rg-np-noedge-n6bsrg-vxj5p9qrl42z" to finish creating: GET https://rp.j2350080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b",
      "name": "83d90b9f-3285-4912-9884-40a78690862b",
      "status": "Failed",
      "startTime": "2026-10-04T00:37:10.755457782Z",
      "endTime": "2026-10-04T00:56:17.299103912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0x2f72d213c320>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-n6bsrg" in resourcegroup="rg-np-noedge-n6bsrg-vxj5p9qrl42z" to finish creating: GET https://rp.j2350080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/83d90b9f-3285-4912-9884-40a78690862b",
      "name": "83d90b9f-3285-4912-9884-40a78690862b",
      "status": "Failed",
      "startTime": "2026-10-04T00:37:10.755457782Z",
      "endTime": "2026-10-04T00:56:17.299103912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-2kn7g to exist\nMachine i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-n6bsrg-vxj5p9qrl42z-np-ne-n6bsrg--managed/providers/Microsoft.Compute/virtualMachines/i4d7j9t8e0g6i1k-npnoedge-4-20-fgjsm-nh6rw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T22:09:35Z
  Ended:       2026-10-03T22:15:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0690944-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T21:07:34Z
  Ended:       2026-10-03T21:13:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1818624-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3528a4bdeac0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vljzr5rw54g9" to finish creating: GET https://rp.j7922048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e",
      "name": "336a4045-fdd2-4a2c-b138-8d99b939169e",
      "status": "Failed",
      "startTime": "2026-10-03T20:19:49.133571808Z",
      "endTime": "2026-10-03T20:38:58.66181942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 3 of 3 machines are not healthy\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 3 of 3 machines are not ready\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3528a4bdeac0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vljzr5rw54g9" to finish creating: GET https://rp.j7922048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/336a4045-fdd2-4a2c-b138-8d99b939169e",
      "name": "336a4045-fdd2-4a2c-b138-8d99b939169e",
      "status": "Failed",
      "startTime": "2026-10-03T20:19:49.133571808Z",
      "endTime": "2026-10-03T20:38:58.66181942Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 3 of 3 machines are not healthy\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 3 of 3 machines are not ready\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-5nr55: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-8xk7g: Creating: virtualmachine creating or updating\nMachine h9f6c5t3m0f9b9x-autoscale-az-bg5l4-kdmwk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T19:53:25Z
  Ended:       2026-10-03T19:59:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7836160-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1590b4a34e00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-hd5zgd7fw6ql" to finish creating: GET https://rp.j0239360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94",
      "name": "0504a146-7d92-49ac-9646-6e1b202d7d94",
      "status": "Failed",
      "startTime": "2026-10-03T18:22:53.322060806Z",
      "endTime": "2026-10-03T18:41:55.898243278Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkgjk-bhjgr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkgjk-bhjgr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1590b4a34e00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-hd5zgd7fw6ql" to finish creating: GET https://rp.j0239360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0504a146-7d92-49ac-9646-6e1b202d7d94",
      "name": "0504a146-7d92-49ac-9646-6e1b202d7d94",
      "status": "Failed",
      "startTime": "2026-10-03T18:22:53.322060806Z",
      "endTime": "2026-10-03T18:41:55.898243278Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkgjk-bhjgr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkgjk-bhjgr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hd5zgd7fw6ql--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkgjk-bhjgr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T18:23:19Z
  Ended:       2026-10-03T18:38:18Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0143360-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-file-csi-driver-operator-56bd98ff47-n9mw7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-file-csi-driver-operator-56bd98ff47-n9mw7 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-disk-csi-driver-operator-68446dbd4d-gp5r7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-disk-csi-driver-operator-68446dbd4d-gp5r7 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-disk-csi-driver-operator-68446dbd4d-72x46", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-disk-csi-driver-operator-68446dbd4d-72x46 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-disk-csi-driver-operator-68446dbd4d-l8w9v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-disk-csi-driver-operator-68446dbd4d-l8w9v has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-03T18:23:01Z
  Ended:       2026-10-03T18:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0143360-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e", pod="azure-file-csi-driver-operator-56bd98ff47-lhdbd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t7fcgvvu6q2gat5neqdkaq9suafasab-y5s6m0p7w4t3q7e/azure-file-csi-driver-operator-56bd98ff47-lhdbd has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xd9700701640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-l9xkxj2bpszs" to finish creating: GET https://rp.j1091968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "name": "d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "status": "Failed",
      "startTime": "2026-10-03T17:56:20.630751478Z",
      "endTime": "2026-10-03T18:15:20.975618436Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xd9700701640>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-l9xkxj2bpszs" to finish creating: GET https://rp.j1091968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "name": "d2ab0eb3-9aa0-4386-9325-4cdb7f9a4b9d",
      "status": "Failed",
      "startTime": "2026-10-03T17:56:20.630751478Z",
      "endTime": "2026-10-03T18:15:20.975618436Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s1f2f0h0t5y3r2c-np-deletable-64c66-chds7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-l9xkxj2bpszs--managed/providers/Microsoft.Compute/virtualMachines/s1f2f0h0t5y3r2c-np-deletable-64c66-chds7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 69 time(s) (63 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-52l2j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-52l2j has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-vdz8t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-vdz8t has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-k2787", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-k2787 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-zp9nc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-zp9nc has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-vk8sg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-vk8sg has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-03T17:36:48Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-fvs76", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-fvs76 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-llstd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-llstd has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-5mcng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-5mcng has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9sqdg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9sqdg has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-03T17:36:57Z
  Ended:       2026-10-03T17:51:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-btfk4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-btfk4 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-dq2vz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-dq2vz has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-x6wz7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-x6wz7 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-ctpt9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-ctpt9 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-j2d9z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-j2d9z has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-4h79j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-4h79j has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-z2w4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-z2w4b has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-nkt4w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-nkt4w has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-ffhbr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-ffhbr has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-csvsv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-csvsv has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-s4h6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-s4h6v has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-dl5l4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-dl5l4 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-47862", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-47862 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-gw8rm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-gw8rm has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-mrcs5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-mrcs5 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-tzs9x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-tzs9x has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-pjk6c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-pjk6c has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-vdtmc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-vdtmc has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-7kmn4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-7kmn4 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-6w4zc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-6w4zc has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-mnfxd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-mnfxd has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-m84qg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-m84qg has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-j8kmn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-j8kmn has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-r827x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-r827x has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lscgk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lscgk has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-gj4v6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-gj4v6 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-2mvqm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-2mvqm has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lw6l6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lw6l6 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9w425", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9w425 has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-ndzg5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-ndzg5 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-b662v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-b662v has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-wj72x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-wj72x has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-5c5cd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-5c5cd has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lkvbz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lkvbz has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-22klt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-22klt has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-rxm4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-rxm4b has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-r7qwh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-r7qwh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-xhl5w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-xhl5w has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-h4p6b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-h4p6b has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-j7f54", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-j7f54 has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-bs5j7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-bs5j7 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-lwcng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-lwcng has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9n44n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9n44n has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-dp4wn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-dp4wn has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-8qlz4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-8qlz4 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-9n2td", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-9n2td has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-d6x6h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-d6x6h has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-03T17:37:51Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-92ktd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-92ktd has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-sxxzw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-sxxzw has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:51:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-zvv2v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-zvv2v has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-2hsfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-2hsfs has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-nz8k8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-nz8k8 has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-gssl7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-gssl7 has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-03T17:37:52Z
  Ended:       2026-10-03T17:56:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1091968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u", pod="kube-controller-manager-5b899d5dd9-7865d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t7en02emtg8ui89m978mkg1haofcv21-x6l3l6y2c5m1t6u/kube-controller-manager-5b899d5dd9-7865d has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x1737a1124260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kjgdvvpr4g6r" to finish creating: GET https://rp.j0472448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a",
      "name": "9760730d-edb7-41c2-8626-62f85b63fc1a",
      "status": "Failed",
      "startTime": "2026-10-03T16:34:46.276618892Z",
      "endTime": "2026-10-03T16:53:53.086174722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x1737a1124260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-kjgdvvpr4g6r" to finish creating: GET https://rp.j0472448.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9760730d-edb7-41c2-8626-62f85b63fc1a",
      "name": "9760730d-edb7-41c2-8626-62f85b63fc1a",
      "status": "Failed",
      "startTime": "2026-10-03T16:34:46.276618892Z",
      "endTime": "2026-10-03T16:53:53.086174722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-kjgdvvpr4g6r--managed/providers/Microsoft.Compute/virtualMachines/n0m4o0c1m0o6o8n-np-1-zf8dz-6rq54 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x22d6ad451998>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x22d6ad451998>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-47mrq.apps.aro.cilium-cluster.iobe.j9578752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T16:03:43Z
  Ended:       2026-10-03T16:09:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9578752-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-vzb878crbjw5" to finish creating: GET https://rp.j0614912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "name": "737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "status": "Failed",
  "startTime": "2026-10-03T14:04:39.339033685Z",
  "endTime": "2026-10-03T14:23:40.256240312Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-vzb878crbjw5" to finish creating: GET https://rp.j0614912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "name": "737b799d-a4dd-42d1-a4a2-9cd870de0e20",
  "status": "Failed",
  "startTime": "2026-10-03T14:04:39.339033685Z",
  "endTime": "2026-10-03T14:23:40.256240312Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vzb878crbjw5--managed/providers/Microsoft.Compute/virtualMachines/e2x8w5d3p9g3r3d-np-update-nodes-mkg2d-mzpcs to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x1d83c5b77a40>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-tzccwkmbcld2" to finish creating: GET https://rp.j4137216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70",
      "name": "63335228-135d-4644-b7d4-651deba91b70",
      "status": "Failed",
      "startTime": "2026-10-03T13:14:48.144915783Z",
      "endTime": "2026-10-03T13:33:57.141188553Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-47kjt-tgj5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-47kjt-tgj5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x1d83c5b77a40>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-tzccwkmbcld2" to finish creating: GET https://rp.j4137216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/63335228-135d-4644-b7d4-651deba91b70",
      "name": "63335228-135d-4644-b7d4-651deba91b70",
      "status": "Failed",
      "startTime": "2026-10-03T13:14:48.144915783Z",
      "endTime": "2026-10-03T13:33:57.141188553Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-47kjt-tgj5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-47kjt-tgj5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-tzccwkmbcld2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-47kjt-tgj5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-10-03T12:18:04.762Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 description="Step swift-vnet\n  Kind: Shell\n  Command: ./swift-vnet.sh\n"
time=2026-10-03T12:18:05.175Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: Failed to load or parse file /tmp/azure-cli-config-3770628587/azureProfile.json. It will be overridden by default settings.\nERROR: The subscription of 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' doesn't exist in cloud 'AzureCloud'.\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2d36ccbc2160>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vgn28b5mgscc" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "name": "1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "status": "Failed",
      "startTime": "2026-10-03T13:01:15.160971572Z",
      "endTime": "2026-10-03T13:20:17.766960003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2d36ccbc2160>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-vgn28b5mgscc" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "name": "1833854b-b6c4-421d-92ac-acbc850a2d4d",
      "status": "Failed",
      "startTime": "2026-10-03T13:01:15.160971572Z",
      "endTime": "2026-10-03T13:20:17.766960003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-6hpdq to exist\nMachine r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vgn28b5mgscc--managed/providers/Microsoft.Compute/virtualMachines/r7b5q6v9r2x8a0b-autoscale-noaz-779kw-r4zl5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x227e9786b20>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-9q2trs" in resourcegroup="rg-np-skip-minor-9q2trs-89955s58px2l" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "name": "8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "status": "Failed",
      "startTime": "2026-10-03T12:59:28.477851902Z",
      "endTime": "2026-10-03T13:18:37.714240199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x227e9786b20>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-9q2trs" in resourcegroup="rg-np-skip-minor-9q2trs-89955s58px2l" to finish creating: GET https://rp.j5049088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "name": "8f9c34cd-a348-440a-9cce-2255b2dba56d",
      "status": "Failed",
      "startTime": "2026-10-03T12:59:28.477851902Z",
      "endTime": "2026-10-03T13:18:37.714240199Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9q2trs-89955s58px2l-np-sm-9q2trs--managed/providers/Microsoft.Compute/virtualMachines/e1y7t9q3u1x3l7k-nps-420-422-6jsw9-hhzvf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T12:40:19Z
  Ended:       2026-10-03T13:01:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5049088-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T10:35:51Z
  Ended:       2026-10-03T10:41:49Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4618112-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1ccede026220>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-74v4krpx78dr" to finish creating: GET https://rp.j7088384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551",
      "name": "61d03357-4fb7-49b2-a777-a99ffa662551",
      "status": "Failed",
      "startTime": "2026-10-03T10:52:21.517435656Z",
      "endTime": "2026-10-03T11:11:28.16247644Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1ccede026220>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-74v4krpx78dr" to finish creating: GET https://rp.j7088384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61d03357-4fb7-49b2-a777-a99ffa662551",
      "name": "61d03357-4fb7-49b2-a777-a99ffa662551",
      "status": "Failed",
      "startTime": "2026-10-03T10:52:21.517435656Z",
      "endTime": "2026-10-03T11:11:28.16247644Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 3 machines are not healthy\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n; node pool condition AllMachinesReady is False: 2 of 3 machines are not ready\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-ct7st to exist\nMachine c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-74v4krpx78dr--managed/providers/Microsoft.Compute/virtualMachines/c0j2z3h4d3u2n8r-autoscale-noaz-5gxpz-nbhvt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T09:10:27Z
  Ended:       2026-10-03T09:28:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1175680-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T09:15:27Z
  Ended:       2026-10-03T09:28:26Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1175680-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T08:23:18Z
  Ended:       2026-10-03T08:30:15Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7356672-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0x1229c1294580>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-q4mxjczq7tsd" to finish creating: GET https://rp.j2783744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080",
      "name": "7757c823-88ba-458e-aed3-9dfdac2d7080",
      "status": "Failed",
      "startTime": "2026-10-03T05:03:30.952777318Z",
      "endTime": "2026-10-03T05:22:34.856876402Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0x1229c1294580>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-q4mxjczq7tsd" to finish creating: GET https://rp.j2783744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7757c823-88ba-458e-aed3-9dfdac2d7080",
      "name": "7757c823-88ba-458e-aed3-9dfdac2d7080",
      "status": "Failed",
      "startTime": "2026-10-03T05:03:30.952777318Z",
      "endTime": "2026-10-03T05:22:34.856876402Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-62mv4-kp8nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-q4mxjczq7tsd--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-62mv4-kp8nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T05:06:36Z
  Ended:       2026-10-03T05:12:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2783744-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x331a5ee39be0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-hzfdbcnlckj2" to finish creating: GET https://rp.j8366592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "name": "f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "status": "Failed",
      "startTime": "2026-10-03T05:21:56.267331547Z",
      "endTime": "2026-10-03T05:41:00.127542846Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x331a5ee39be0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-hzfdbcnlckj2" to finish creating: GET https://rp.j8366592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "name": "f7eb0be6-172b-405f-8019-aa4a01912e9a",
      "status": "Failed",
      "startTime": "2026-10-03T05:21:56.267331547Z",
      "endTime": "2026-10-03T05:41:00.127542846Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-hzfdbcnlckj2--managed/providers/Microsoft.Compute/virtualMachines/f3y4c6b9n5p5z6y-cordon-np-qb8hf-8gc5p to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T04:38:45Z
  Ended:       2026-10-03T04:44:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3883264-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-03T04:27:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j3883264-svc", component="backend", name="openshiftversioncatalog", region="westus3", severity="warning"
  Description: Backend controller workqueue openshiftversioncatalog has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0x1113bcb9c160>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-rckbfw" in resourcegroup="rg-np-downgrade-rckbfw-856zsqvm5rkb" to finish creating: GET https://rp.j7360256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4",
      "name": "074aacd3-7322-43b0-9305-f06220347ff4",
      "status": "Failed",
      "startTime": "2026-10-03T04:27:23.785116321Z",
      "endTime": "2026-10-03T04:46:25.26888513Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0x1113bcb9c160>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-rckbfw" in resourcegroup="rg-np-downgrade-rckbfw-856zsqvm5rkb" to finish creating: GET https://rp.j7360256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/074aacd3-7322-43b0-9305-f06220347ff4",
      "name": "074aacd3-7322-43b0-9305-f06220347ff4",
      "status": "Failed",
      "startTime": "2026-10-03T04:27:23.785116321Z",
      "endTime": "2026-10-03T04:46:25.26888513Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-rckbfw-856zsqvm5rkb-np-dg-rckbfw--managed/providers/Microsoft.Compute/virtualMachines/l7r8a0n5m4k8l1t-npdg-4-21-z7xhd-fq8sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T04:07:22Z
  Ended:       2026-10-03T04:13:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7360256-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T03:07:49Z
  Ended:       2026-10-03T03:19:50Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1034112-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T03:08:45Z
  Ended:       2026-10-03T03:22:45Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1034112-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3263781c03a0>: 
    DNS for route host agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3263781c03a0>: 
    DNS for route host agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j98xq.apps.aro.v20261001.9je8.j4629376.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T01:56:32Z
  Ended:       2026-10-03T02:03:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4629376-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T01:57:33Z
  Ended:       2026-10-03T02:10:32Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4629376-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-xt2c7s"
Unexpected error:
    <*fmt.wrapError | 0x1b2c75ad8a80>: 
    DNS for route host agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-xt2c7s"
Unexpected error:
    <*fmt.wrapError | 0x1b2c75ad8a80>: 
    DNS for route host agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-c8pn7.apps.aro.s8a7u2c4b6z3h8o.rc93.j6512256.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T02:01:45Z
  Ended:       2026-10-03T02:07:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6512256-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-10-03T01:35:28.208Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-10-03T01:35:28.778Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy values="map[azureArmHelperIdentityCertName:armHelperPoolCert-10 azureArmHelperIdentityClientId:0a9e3940-6bc3-4cbf-a8f7-c65c3d8d2462 azureArmHelperMockFpaPrincipalId:47f69502-0065-4d9a-b19b-d403e183d2f4 azureCloudEnvironmentName:AzurePublicCloud azureCsMiClientId:de4f7be2-47e3-4002-adc2-77c2fcdec830 azureFirstPartyApplicationClientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX azureMiMockServicePrincipalCertName:msiMockPoolCert-17 azureMiMockServicePrincipalClientId:e7112f7f-5b45-480e-bc44-abf9f219e6f0 azureMiMockServicePrincipalPrincipalId:39eefc4f-5e24-491d-9f09-bcc0ad573bcf azureOperatorsMI:map[dev:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] public:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] roleSetName:dev] backplaneURL: batchProcesses: batchProcessesDryRun:true clientId:foo clientScopes:openid clientSecret:bar clusterErrorReport:false cosmosDatabaseName:arohcpci01-rp-j4407168 cosmosDatabaseUrl:https://arohcpci01-rp-j4407168.documents.azure.com:443/ cpuRequest:630m csDeploymentStrategy:map[rollingUpdate:map[maxSurge:1 maxUnavailable:0] type:RollingUpdate] dataPlaneHAProxyImage:arohcpocpdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 databaseAuthMethod:postgres databaseDisableTls:true databaseHost:ocm-cs-db databaseK8sSecretName:ocm-cs-db databaseName:ocm-cs-db databasePassword:TheBlurstOfTimes databasePort:5432 databaseServiceName:ocm-cs-db databaseUser:ocm databaseVolumeCapacity:512Mi debugMaxDuration:5m debugPort: defaultExpiration:0 deployment:map[zoneCount:3] environment:arohcpci01 forceMigration: fpaCertName:firstPartyCert2 gatewayUrl:http://127.0.0.1:9090 global:map[imageDigest:sha256:5fffce295f8b49be38ae2385714d09af29700dd3dc60c40a058f7aeb66e97d72] imageDigest:sha256:5fffce295f8b49be38ae2385714d09af29700dd3dc60c40a058f7aeb66e97d72 imageRegistry:arohcpsvcdev.azurecr.io imageRepository:app-sre/aro-hcp-clusters-service insecure:false jwksUrl:http://localhost logLevel:4 machinePoolMigrationWorkerPeriod:1h managedIdentitiesDataPlaneAudienceResource:https://dummy.org maximumExpiration:0 memoryLimit:unlimited memoryRequest:150Mi ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev ocpAcrResourceUrl:arohcpocpdev.azurecr.io oidcIssuerBaseUrl:https://arohcpci01oidcj4407168.z9.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj4407168.blob.core.windows.net/ postgresqlVersion:12 provisionShardClusterLimit:100 pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:app-sre/aro-hcp-clusters-service:pull workloadIdentityClientId:27f533be-4c23-4032-9d19-da9c7454bd3b workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] region:canadacentral replicas:3 runtimeMode:aro-hcp serviceAccountName:clusters-service serviceKeyVaultName:aro-hcp-dev-svc-kv tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesEnabled:true tlsCertificatesIssuer:Self tokenUrl:http://localhost useAzureDB:false userDefinedDnsBaseDomain:i1.devshift.org]"
time=2026-10-03T01:35:28.779Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-03T01:35:29.089Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy namespace=clusters-service
time=2026-10-03T01:35:29.415Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy dryRun=false
time=2026-10-03T01:35:29.416Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-03T01:35:29.465Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-03T01:45:35.236Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-10-03T01:45:35.237Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-03T01:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4407168-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-74695d59f9-hxdpn", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-74695d59f9-hxdpn has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-03T01:42:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4407168-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-74695d59f9-bfnvs", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-74695d59f9-bfnvs has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-03T01:42:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4407168-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-74695d59f9-wjv8w", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-74695d59f9-wjv8w has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-h26qrg" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0x34bdd7d932c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-5fxfbtp77wdj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-h26qrg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-h26qrg" to 5.1.0-0.nightly-multi-2026-09-30-181926
Unexpected error:
    <*exported.ResponseError | 0x34bdd7d932c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-5fxfbtp77wdj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-h26qrg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x156bbec7cb0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-48mh6dg9zlr8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x156bbec7cb0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-48mh6dg9zlr8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x210c33ec90b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8fsbbrnx57fx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x210c33ec90b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8fsbbrnx57fx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xd97df9bad80>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-cskpwn77dkwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xd97df9bad80>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-cskpwn77dkwd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-4kgdcw" for cluster "cluster-nightly-4-20-4kgdcw"
Unexpected error:
    <*fmt.wrapError | 0x781abe80140>: 
    failed to create NodePool np-4kgdcw: failed starting nodepool creation "np-4kgdcw" for cluster "cluster-nightly-4-20-4kgdcw" in resourcegroup="rg-nightly-4-20-4t8l44cr8nbr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-4t8l44cr8nbr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-4kgdcw/nodePools/np-4kgdcw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-4kgdcw" for cluster "cluster-nightly-4-20-4kgdcw"
Unexpected error:
    <*fmt.wrapError | 0x781abe80140>: 
    failed to create NodePool np-4kgdcw: failed starting nodepool creation "np-4kgdcw" for cluster "cluster-nightly-4-20-4kgdcw" in resourcegroup="rg-nightly-4-20-4t8l44cr8nbr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-4t8l44cr8nbr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-4kgdcw/nodePools/np-4kgdcw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2d7c17881b00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-hn22fz5qv7wv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2d7c17881b00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-hn22fz5qv7wv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:241]: failed to verify HCP cluster viability
Unexpected error:
    <*errors.joinError | 0x29a4ab6ac030>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:241]: failed to verify HCP cluster viability
Unexpected error:
    <*errors.joinError | 0x29a4ab6ac030>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0x31d5efc20c00>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-b4wh48" in resourcegroup="rg-np-version-upgrade-b4wh48-bzk7j574q45w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b4wh48-bzk7j574q45w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-b4wh48/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0x31d5efc20c00>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-b4wh48" in resourcegroup="rg-np-version-upgrade-b4wh48-bzk7j574q45w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b4wh48-bzk7j574q45w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-b4wh48/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-54csls" to 4.21.0-0.nightly-multi-2026-10-02-115050
Unexpected error:
    <*exported.ResponseError | 0x2086109860c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-hx8x6fq8plbx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-54csls
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-54csls" to 4.21.0-0.nightly-multi-2026-10-02-115050
Unexpected error:
    <*exported.ResponseError | 0x2086109860c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-hx8x6fq8plbx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-54csls
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x3cb7bb3ff960>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-pf8r4pxpb5hl": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-pf8r4pxpb5hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x3cb7bb3ff960>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-pf8r4pxpb5hl": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-pf8r4pxpb5hl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xcf8828ab170>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-crv2w79vmlxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xcf8828ab170>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-crv2w79vmlxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x325c79dc0060>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-987nc9dd4vhq": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-987nc9dd4vhq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x325c79dc0060>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-987nc9dd4vhq": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-987nc9dd4vhq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2861f94db110>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-s8tjmf2mg47r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a70c1239-194d-44ad-b66c-18f11327f9a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a70c1239-194d-44ad-b66c-18f11327f9a4",
      "name": "a70c1239-194d-44ad-b66c-18f11327f9a4",
      "status": "Failed",
      "startTime": "2026-10-03T09:45:52.709942431Z",
      "endTime": "2026-10-03T10:04:55.390706618Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-mf948-nwnch: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-s8tjmf2mg47r--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-mf948-nwnch to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-mf948-nwnch: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-s8tjmf2mg47r--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-mf948-nwnch to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2861f94db110>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-s8tjmf2mg47r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a70c1239-194d-44ad-b66c-18f11327f9a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a70c1239-194d-44ad-b66c-18f11327f9a4",
      "name": "a70c1239-194d-44ad-b66c-18f11327f9a4",
      "status": "Failed",
      "startTime": "2026-10-03T09:45:52.709942431Z",
      "endTime": "2026-10-03T10:04:55.390706618Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-mf948-nwnch: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-s8tjmf2mg47r--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-mf948-nwnch to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-mf948-nwnch: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-s8tjmf2mg47r--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-mf948-nwnch to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x33e8cc763a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-wrvrktrj9pn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a021811c-f25e-4fdd-b88c-cc64a68c8188
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a021811c-f25e-4fdd-b88c-cc64a68c8188",
      "name": "a021811c-f25e-4fdd-b88c-cc64a68c8188",
      "status": "Failed",
      "startTime": "2026-10-03T09:41:16.318835994Z",
      "endTime": "2026-10-03T10:00:22.618172935Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k3u6n5v9b5f7j2v-np-1-9khfh-nvt5s: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k3u6n5v9b5f7j2v-np-1-9khfh-nvt5s: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x33e8cc763a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-wrvrktrj9pn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a021811c-f25e-4fdd-b88c-cc64a68c8188
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/a021811c-f25e-4fdd-b88c-cc64a68c8188",
      "name": "a021811c-f25e-4fdd-b88c-cc64a68c8188",
      "status": "Failed",
      "startTime": "2026-10-03T09:41:16.318835994Z",
      "endTime": "2026-10-03T10:00:22.618172935Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k3u6n5v9b5f7j2v-np-1-9khfh-nvt5s: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k3u6n5v9b5f7j2v-np-1-9khfh-nvt5s: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a380b9ff820>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jdf825hs8fn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/1ffbe111-707b-4771-8ddd-d413913d8bde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/1ffbe111-707b-4771-8ddd-d413913d8bde",
      "name": "1ffbe111-707b-4771-8ddd-d413913d8bde",
      "status": "Failed",
      "startTime": "2026-10-03T10:22:50.222769152Z",
      "endTime": "2026-10-03T10:41:50.376552801Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jdf825hs8fn9--managed/providers/Microsoft.Compute/virtualMachines/f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jdf825hs8fn9--managed/providers/Microsoft.Compute/virtualMachines/f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a380b9ff820>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jdf825hs8fn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/1ffbe111-707b-4771-8ddd-d413913d8bde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/1ffbe111-707b-4771-8ddd-d413913d8bde",
      "name": "1ffbe111-707b-4771-8ddd-d413913d8bde",
      "status": "Failed",
      "startTime": "2026-10-03T10:22:50.222769152Z",
      "endTime": "2026-10-03T10:41:50.376552801Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jdf825hs8fn9--managed/providers/Microsoft.Compute/virtualMachines/f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jdf825hs8fn9--managed/providers/Microsoft.Compute/virtualMachines/f0m1e1q3s5q9h4n-np-deletable-cb4h4-4ccd6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-v6zlk7" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x3aa50a408f30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-7fls5z997c4n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-v6zlk7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-v6zlk7" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x3aa50a408f30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-7fls5z997c4n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-v6zlk7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-msl8ps" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x21a7a658f5c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-l7wmbxqck8gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-msl8ps
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-msl8ps" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x21a7a658f5c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-l7wmbxqck8gf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-msl8ps
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-9hrm6z" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1415c20a5500>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-cck68gj6qfsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9hrm6z
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-9hrm6z" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1415c20a5500>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-cck68gj6qfsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9hrm6z
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x61296253c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-96mcz9ktg57h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x61296253c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-96mcz9ktg57h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-x4v9ss" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x5a0752a7c50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-9j7fg4k8csln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-x4v9ss
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-x4v9ss" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x5a0752a7c50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-9j7fg4k8csln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-x4v9ss
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1ed03c0ba1c0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-nrr89q" resourcegroup="rg-np-dg-minor-nrr89q-d5btp9lwtkwk": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-nrr89q-d5btp9lwtkwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-nrr89q/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1ed03c0ba1c0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-nrr89q" resourcegroup="rg-np-dg-minor-nrr89q-d5btp9lwtkwk": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-nrr89q-d5btp9lwtkwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-nrr89q/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-w78qcb" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x39bf51e7ba70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-l8ksqxqxvhll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-w78qcb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-w78qcb" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x39bf51e7ba70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-l8ksqxqxvhll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-w78qcb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x6bee01cdac0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-6td8ds" resourcegroup="rg-np-skip-minor-6td8ds-d2dvtglcn2l2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-6td8ds-d2dvtglcn2l2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-6td8ds/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x6bee01cdac0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-6td8ds" resourcegroup="rg-np-skip-minor-6td8ds-d2dvtglcn2l2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-6td8ds-d2dvtglcn2l2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-6td8ds/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x177ce192fa20>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-z4pwtb" resourcegroup="rg-np-version-upgrade-z4pwtb-4qn5bp5csjzb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z4pwtb-4qn5bp5csjzb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-z4pwtb/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x177ce192fa20>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-z4pwtb" resourcegroup="rg-np-version-upgrade-z4pwtb-4qn5bp5csjzb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z4pwtb-4qn5bp5csjzb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-z4pwtb/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x143a1310b240>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-rc8d5d" resourcegroup="rg-np-version-upgrade-rc8d5d-w2t72x2zshww": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-rc8d5d-w2t72x2zshww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-rc8d5d/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x143a1310b240>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-rc8d5d" resourcegroup="rg-np-version-upgrade-rc8d5d-w2t72x2zshww": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-rc8d5d-w2t72x2zshww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-rc8d5d/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-w2grzz" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x30efed875560>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-52rk6bpxrc9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-w2grzz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-w2grzz" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x30efed875560>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-52rk6bpxrc9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-w2grzz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x148c3b6e47b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tvg824p96xln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x148c3b6e47b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tvg824p96xln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x95d711c6660>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fqfljmzwnn7j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x95d711c6660>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fqfljmzwnn7j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x16c8ce6032a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-4rvxbgx64wn7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4rvxbgx64wn7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x16c8ce6032a0>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-4rvxbgx64wn7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-4rvxbgx64wn7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-2bw2nd" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x3e7c72aaef90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-h5wrz4q9tk28/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-2bw2nd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-2bw2nd" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x3e7c72aaef90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-h5wrz4q9tk28/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-2bw2nd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-4c45hm" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x1c31b53c8540>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-7jn7lrmbjhwn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-4c45hm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-4c45hm" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x1c31b53c8540>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-7jn7lrmbjhwn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-4c45hm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x18d8f5386060>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-xvr7qd" resourcegroup="rg-np-noedge-xvr7qd-fdgmzp5mlmp2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xvr7qd-fdgmzp5mlmp2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-xvr7qd/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x18d8f5386060>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-xvr7qd" resourcegroup="rg-np-noedge-xvr7qd-fdgmzp5mlmp2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xvr7qd-fdgmzp5mlmp2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-xvr7qd/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xa952f808ee0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-x47s9n" resourcegroup="rg-np-downgrade-x47s9n-djktngm9cgxt": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-x47s9n-djktngm9cgxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-x47s9n/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xa952f808ee0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-x47s9n" resourcegroup="rg-np-downgrade-x47s9n-djktngm9cgxt": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-x47s9n-djktngm9cgxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-x47s9n/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-95bqxj" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x1aa712d69710>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-b5g72gdsstzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-95bqxj
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-95bqxj" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x1aa712d69710>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-b5g72gdsstzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-95bqxj
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0xc0cb099f2c0>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-cvfvg5" resourcegroup="rg-np-version-upgrade-cvfvg5-r74zmmc476nw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-cvfvg5-r74zmmc476nw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-cvfvg5/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0xc0cb099f2c0>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-cvfvg5" resourcegroup="rg-np-version-upgrade-cvfvg5-r74zmmc476nw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-cvfvg5-r74zmmc476nw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-cvfvg5/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x110fe0e8c640>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-djddxfxjp2wr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-djddxfxjp2wr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x110fe0e8c640>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-djddxfxjp2wr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-djddxfxjp2wr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-79cvw2" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x23c456aecc90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-8lpbkjqgxtmb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-79cvw2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-79cvw2" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x23c456aecc90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-8lpbkjqgxtmb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-79cvw2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x3731739b0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4h58pvg4tw2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1659164e-7252-4678-9399-950feff9f29d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1659164e-7252-4678-9399-950feff9f29d",
      "name": "1659164e-7252-4678-9399-950feff9f29d",
      "status": "Failed",
      "startTime": "2026-10-03T23:15:46.648351931Z",
      "endTime": "2026-10-03T23:34:50.305129886Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-hkqzd-6mfw6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-hkqzd-7x9wt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-hkqzd-6mfw6: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-hkqzd-7x9wt: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x3731739b0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4h58pvg4tw2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1659164e-7252-4678-9399-950feff9f29d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1659164e-7252-4678-9399-950feff9f29d",
      "name": "1659164e-7252-4678-9399-950feff9f29d",
      "status": "Failed",
      "startTime": "2026-10-03T23:15:46.648351931Z",
      "endTime": "2026-10-03T23:34:50.305129886Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-hkqzd-6mfw6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-hkqzd-7x9wt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-hkqzd-6mfw6: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-hkqzd-7x9wt: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zb9djq" for cluster "cluster-nightly-4-20-zb9djq"
Unexpected error:
    <*fmt.wrapError | 0x242124721260>: 
    failed to create NodePool np-zb9djq: failed starting nodepool creation "np-zb9djq" for cluster "cluster-nightly-4-20-zb9djq" in resourcegroup="rg-nightly-4-20-r55xk7zmjr9c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-r55xk7zmjr9c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zb9djq/nodePools/np-zb9djq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zb9djq" for cluster "cluster-nightly-4-20-zb9djq"
Unexpected error:
    <*fmt.wrapError | 0x242124721260>: 
    failed to create NodePool np-zb9djq: failed starting nodepool creation "np-zb9djq" for cluster "cluster-nightly-4-20-zb9djq" in resourcegroup="rg-nightly-4-20-r55xk7zmjr9c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-r55xk7zmjr9c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-zb9djq/nodePools/np-zb9djq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0x184038744400>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-tzvzfh" in resourcegroup="rg-np-version-upgrade-tzvzfh-8sk9jqkdk6gz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tzvzfh-8sk9jqkdk6gz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tzvzfh/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0x184038744400>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-tzvzfh" in resourcegroup="rg-np-version-upgrade-tzvzfh-8sk9jqkdk6gz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tzvzfh-8sk9jqkdk6gz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tzvzfh/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x366855210620>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4c8kl482c2st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37548aa4-a4e7-458d-b686-fb1201996386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37548aa4-a4e7-458d-b686-fb1201996386",
      "name": "37548aa4-a4e7-458d-b686-fb1201996386",
      "status": "Failed",
      "startTime": "2026-10-03T23:15:28.253854505Z",
      "endTime": "2026-10-03T23:34:30.095088976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-n2nhw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-xsvvt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-n2nhw: Creating: virtualmachine creating or updating\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-xsvvt: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x366855210620>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4c8kl482c2st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37548aa4-a4e7-458d-b686-fb1201996386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37548aa4-a4e7-458d-b686-fb1201996386",
      "name": "37548aa4-a4e7-458d-b686-fb1201996386",
      "status": "Failed",
      "startTime": "2026-10-03T23:15:28.253854505Z",
      "endTime": "2026-10-03T23:34:30.095088976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-n2nhw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-xsvvt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-n2nhw: Creating: virtualmachine creating or updating\nMachine c4o5k1a3s4v8w4p-np-1-gjcb5-xsvvt: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dz68ws" for cluster "cluster-nightly-5-0-dz68ws"
Unexpected error:
    <*fmt.wrapError | 0x31a06c9a5d00>: 
    failed to create NodePool np-dz68ws: failed waiting for nodepool="np-dz68ws" for cluster "cluster-nightly-5-0-dz68ws" in resourcegroup="rg-nightly-5-0-4t5m8n4czrvj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2e4aeefa-8e4c-446e-bbb2-afff95c12936
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2e4aeefa-8e4c-446e-bbb2-afff95c12936",
      "name": "2e4aeefa-8e4c-446e-bbb2-afff95c12936",
      "status": "Failed",
      "startTime": "2026-10-03T23:35:55.566084896Z",
      "endTime": "2026-10-03T23:55:00.222357543Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-95fl8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-kc94l: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-95fl8: Creating: virtualmachine creating or updating\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-kc94l: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dz68ws" for cluster "cluster-nightly-5-0-dz68ws"
Unexpected error:
    <*fmt.wrapError | 0x31a06c9a5d00>: 
    failed to create NodePool np-dz68ws: failed waiting for nodepool="np-dz68ws" for cluster "cluster-nightly-5-0-dz68ws" in resourcegroup="rg-nightly-5-0-4t5m8n4czrvj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2e4aeefa-8e4c-446e-bbb2-afff95c12936
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2e4aeefa-8e4c-446e-bbb2-afff95c12936",
      "name": "2e4aeefa-8e4c-446e-bbb2-afff95c12936",
      "status": "Failed",
      "startTime": "2026-10-03T23:35:55.566084896Z",
      "endTime": "2026-10-03T23:55:00.222357543Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-95fl8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-kc94l: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-95fl8: Creating: virtualmachine creating or updating\nMachine h3s7g6c6v2v9l8v-np-dz68ws-76lrt-kc94l: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x36252fbe4930>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-548xlmq4vp84/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x36252fbe4930>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-548xlmq4vp84/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3102b57c8300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-5xj4fb5fc6dl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3102b57c8300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-5xj4fb5fc6dl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vgspf2" for cluster "cluster-nightly-4-21-vgspf2"
Unexpected error:
    <*fmt.wrapError | 0x34ef387fcea0>: 
    failed to create NodePool np-vgspf2: failed waiting for nodepool="np-vgspf2" for cluster "cluster-nightly-4-21-vgspf2" in resourcegroup="rg-nightly-4-21-dsfs76js9854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78a87535-b0bc-47d2-9812-340ab7a0fb8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78a87535-b0bc-47d2-9812-340ab7a0fb8e",
      "name": "78a87535-b0bc-47d2-9812-340ab7a0fb8e",
      "status": "Failed",
      "startTime": "2026-10-03T23:34:31.068400483Z",
      "endTime": "2026-10-03T23:53:40.358231747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-dh299: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-jlf82: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-dh299: Creating: virtualmachine creating or updating\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-jlf82: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vgspf2" for cluster "cluster-nightly-4-21-vgspf2"
Unexpected error:
    <*fmt.wrapError | 0x34ef387fcea0>: 
    failed to create NodePool np-vgspf2: failed waiting for nodepool="np-vgspf2" for cluster "cluster-nightly-4-21-vgspf2" in resourcegroup="rg-nightly-4-21-dsfs76js9854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78a87535-b0bc-47d2-9812-340ab7a0fb8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78a87535-b0bc-47d2-9812-340ab7a0fb8e",
      "name": "78a87535-b0bc-47d2-9812-340ab7a0fb8e",
      "status": "Failed",
      "startTime": "2026-10-03T23:34:31.068400483Z",
      "endTime": "2026-10-03T23:53:40.358231747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-dh299: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-jlf82: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-dh299: Creating: virtualmachine creating or updating\nMachine r3x3m3o0p2b4e4x-np-vgspf2-qj7bv-jlf82: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-4pbb46" for cluster "cluster-nightly-5-1-4pbb46"
Unexpected error:
    <*fmt.wrapError | 0x1fb215239360>: 
    failed to create NodePool np-4pbb46: failed waiting for nodepool="np-4pbb46" for cluster "cluster-nightly-5-1-4pbb46" in resourcegroup="rg-nightly-5-1-4rftrv2tqf7l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e74d3e2c-deeb-4d38-9a72-e8d6c34c576d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e74d3e2c-deeb-4d38-9a72-e8d6c34c576d",
      "name": "e74d3e2c-deeb-4d38-9a72-e8d6c34c576d",
      "status": "Failed",
      "startTime": "2026-10-03T23:37:12.955436587Z",
      "endTime": "2026-10-03T23:56:20.22170931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-4s8vt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-rvt6q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-4s8vt: Creating: virtualmachine creating or updating\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-rvt6q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-4pbb46" for cluster "cluster-nightly-5-1-4pbb46"
Unexpected error:
    <*fmt.wrapError | 0x1fb215239360>: 
    failed to create NodePool np-4pbb46: failed waiting for nodepool="np-4pbb46" for cluster "cluster-nightly-5-1-4pbb46" in resourcegroup="rg-nightly-5-1-4rftrv2tqf7l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e74d3e2c-deeb-4d38-9a72-e8d6c34c576d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e74d3e2c-deeb-4d38-9a72-e8d6c34c576d",
      "name": "e74d3e2c-deeb-4d38-9a72-e8d6c34c576d",
      "status": "Failed",
      "startTime": "2026-10-03T23:37:12.955436587Z",
      "endTime": "2026-10-03T23:56:20.22170931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-4s8vt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-rvt6q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-4s8vt: Creating: virtualmachine creating or updating\nMachine d6a2h5t1r0s8k0n-np-4pbb46-vntxw-rvt6q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x2f86d9c91d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-4kkvfds7xktp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x2f86d9c91d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-4kkvfds7xktp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x145ed3760d50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cxtlbfhqtdd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x145ed3760d50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cxtlbfhqtdd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x104897805d20>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r8db5d4ldfmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12e347fe-1273-4e0c-9b2a-1c191d6d4055
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12e347fe-1273-4e0c-9b2a-1c191d6d4055",
      "name": "12e347fe-1273-4e0c-9b2a-1c191d6d4055",
      "status": "Failed",
      "startTime": "2026-10-03T23:41:40.611562032Z",
      "endTime": "2026-10-04T00:00:46.799966407Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-hlwvx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-xbp6q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-hlwvx: Creating: virtualmachine creating or updating\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-xbp6q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x104897805d20>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r8db5d4ldfmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12e347fe-1273-4e0c-9b2a-1c191d6d4055
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12e347fe-1273-4e0c-9b2a-1c191d6d4055",
      "name": "12e347fe-1273-4e0c-9b2a-1c191d6d4055",
      "status": "Failed",
      "startTime": "2026-10-03T23:41:40.611562032Z",
      "endTime": "2026-10-04T00:00:46.799966407Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-hlwvx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-xbp6q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-hlwvx: Creating: virtualmachine creating or updating\nMachine o0s0h4b0m0k9t7l-np-delete-main-sj9cl-xbp6q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x336e50b8a360>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-njsbz992h4xx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x336e50b8a360>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-njsbz992h4xx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-hz7c47" for cluster "cluster-nightly-4-23-hz7c47"
Unexpected error:
    <*fmt.wrapError | 0x33bfacf01ac0>: 
    failed to create NodePool np-hz7c47: failed waiting for nodepool="np-hz7c47" for cluster "cluster-nightly-4-23-hz7c47" in resourcegroup="rg-nightly-4-23-zskzh2wkzsf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ea1a45b2-8ea6-4372-9874-0ab41e3da185
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ea1a45b2-8ea6-4372-9874-0ab41e3da185",
      "name": "ea1a45b2-8ea6-4372-9874-0ab41e3da185",
      "status": "Failed",
      "startTime": "2026-10-03T23:36:57.511630158Z",
      "endTime": "2026-10-03T23:56:00.189645328Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-hmv9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-qxrgm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-hmv9g: Creating: virtualmachine creating or updating\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-qxrgm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-hz7c47" for cluster "cluster-nightly-4-23-hz7c47"
Unexpected error:
    <*fmt.wrapError | 0x33bfacf01ac0>: 
    failed to create NodePool np-hz7c47: failed waiting for nodepool="np-hz7c47" for cluster "cluster-nightly-4-23-hz7c47" in resourcegroup="rg-nightly-4-23-zskzh2wkzsf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ea1a45b2-8ea6-4372-9874-0ab41e3da185
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ea1a45b2-8ea6-4372-9874-0ab41e3da185",
      "name": "ea1a45b2-8ea6-4372-9874-0ab41e3da185",
      "status": "Failed",
      "startTime": "2026-10-03T23:36:57.511630158Z",
      "endTime": "2026-10-03T23:56:00.189645328Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-hmv9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-qxrgm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-hmv9g: Creating: virtualmachine creating or updating\nMachine p8n6w8u1n2s0m9t-np-hz7c47-xs76h-qxrgm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x1634a92060e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-5tmnm5zpfp6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2872059a-968a-400a-8256-03392979308f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2872059a-968a-400a-8256-03392979308f",
      "name": "2872059a-968a-400a-8256-03392979308f",
      "status": "Failed",
      "startTime": "2026-10-03T02:59:17.985754909Z",
      "endTime": "2026-10-03T03:18:20.080261329Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-x5mt5-q58ts: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-x5mt5-q58ts: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x1634a92060e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-5tmnm5zpfp6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2872059a-968a-400a-8256-03392979308f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2872059a-968a-400a-8256-03392979308f",
      "name": "2872059a-968a-400a-8256-03392979308f",
      "status": "Failed",
      "startTime": "2026-10-03T02:59:17.985754909Z",
      "endTime": "2026-10-03T03:18:20.080261329Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-x5mt5-q58ts: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-x5mt5-q58ts: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1c6f23908030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ddfqq694gxhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bea67035-4bd4-49aa-97d8-eb0a71fb82dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bea67035-4bd4-49aa-97d8-eb0a71fb82dc",
      "name": "bea67035-4bd4-49aa-97d8-eb0a71fb82dc",
      "status": "Failed",
      "startTime": "2026-10-03T02:59:14.55416176Z",
      "endTime": "2026-10-03T03:18:19.785989613Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-7d6m6-8kqhg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-7d6m6-ptnv4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-7d6m6-8kqhg: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-7d6m6-ptnv4: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1c6f23908030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ddfqq694gxhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bea67035-4bd4-49aa-97d8-eb0a71fb82dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bea67035-4bd4-49aa-97d8-eb0a71fb82dc",
      "name": "bea67035-4bd4-49aa-97d8-eb0a71fb82dc",
      "status": "Failed",
      "startTime": "2026-10-03T02:59:14.55416176Z",
      "endTime": "2026-10-03T03:18:19.785989613Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-7d6m6-8kqhg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-7d6m6-ptnv4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-7d6m6-8kqhg: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-7d6m6-ptnv4: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wgvn44" for cluster "cluster-candidate-5-1-wgvn44"
Unexpected error:
    <*fmt.wrapError | 0x582bdd8e0c0>: 
    failed to create NodePool np-wgvn44: failed waiting for nodepool="np-wgvn44" for cluster "cluster-candidate-5-1-wgvn44" in resourcegroup="rg-candidate-5-1-g5g8pnhdz6gx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7b96711e-b171-48d1-87d7-1f055025bc87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7b96711e-b171-48d1-87d7-1f055025bc87",
      "name": "7b96711e-b171-48d1-87d7-1f055025bc87",
      "status": "Failed",
      "startTime": "2026-10-03T03:24:01.084946017Z",
      "endTime": "2026-10-03T03:43:03.38976371Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-5mgvm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-vkmcn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-5mgvm: Creating: virtualmachine creating or updating\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-vkmcn: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wgvn44" for cluster "cluster-candidate-5-1-wgvn44"
Unexpected error:
    <*fmt.wrapError | 0x582bdd8e0c0>: 
    failed to create NodePool np-wgvn44: failed waiting for nodepool="np-wgvn44" for cluster "cluster-candidate-5-1-wgvn44" in resourcegroup="rg-candidate-5-1-g5g8pnhdz6gx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7b96711e-b171-48d1-87d7-1f055025bc87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7b96711e-b171-48d1-87d7-1f055025bc87",
      "name": "7b96711e-b171-48d1-87d7-1f055025bc87",
      "status": "Failed",
      "startTime": "2026-10-03T03:24:01.084946017Z",
      "endTime": "2026-10-03T03:43:03.38976371Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-5mgvm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-vkmcn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-5mgvm: Creating: virtualmachine creating or updating\nMachine j6u5b3c2z9q1l8v-np-wgvn44-b9dp7-vkmcn: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-994gtb" for cluster "cluster-candidate-5-0-994gtb"
Unexpected error:
    <*fmt.wrapError | 0x3b59f6a200a0>: 
    failed to create NodePool np-994gtb: failed waiting for nodepool="np-994gtb" for cluster "cluster-candidate-5-0-994gtb" in resourcegroup="rg-candidate-5-0-dpqs94628k6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3ae09fa-a5f4-44d2-9f00-68063627145c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3ae09fa-a5f4-44d2-9f00-68063627145c",
      "name": "f3ae09fa-a5f4-44d2-9f00-68063627145c",
      "status": "Failed",
      "startTime": "2026-10-03T03:22:48.672584866Z",
      "endTime": "2026-10-03T03:41:53.397092955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-9xspx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-v8vvg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-9xspx: Creating: virtualmachine creating or updating\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-v8vvg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-994gtb" for cluster "cluster-candidate-5-0-994gtb"
Unexpected error:
    <*fmt.wrapError | 0x3b59f6a200a0>: 
    failed to create NodePool np-994gtb: failed waiting for nodepool="np-994gtb" for cluster "cluster-candidate-5-0-994gtb" in resourcegroup="rg-candidate-5-0-dpqs94628k6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3ae09fa-a5f4-44d2-9f00-68063627145c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3ae09fa-a5f4-44d2-9f00-68063627145c",
      "name": "f3ae09fa-a5f4-44d2-9f00-68063627145c",
      "status": "Failed",
      "startTime": "2026-10-03T03:22:48.672584866Z",
      "endTime": "2026-10-03T03:41:53.397092955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-9xspx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-v8vvg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-9xspx: Creating: virtualmachine creating or updating\nMachine w0a5p3m7x2i2z6g-np-994gtb-827pv-v8vvg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zsxxnl" for cluster "cluster-candidate-4-22-zsxxnl"
Unexpected error:
    <*fmt.wrapError | 0x8479408b140>: 
    failed to create NodePool np-zsxxnl: failed waiting for nodepool="np-zsxxnl" for cluster "cluster-candidate-4-22-zsxxnl" in resourcegroup="rg-candidate-4-22-7822rkz4qt8d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fd0879a-ebbb-4fea-b214-ece8494edab0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fd0879a-ebbb-4fea-b214-ece8494edab0",
      "name": "9fd0879a-ebbb-4fea-b214-ece8494edab0",
      "status": "Failed",
      "startTime": "2026-10-03T03:22:25.37619508Z",
      "endTime": "2026-10-03T03:41:30.020299253Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-6hmr2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-b6wrl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-6hmr2: Creating: virtualmachine creating or updating\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-b6wrl: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-zsxxnl" for cluster "cluster-candidate-4-22-zsxxnl"
Unexpected error:
    <*fmt.wrapError | 0x8479408b140>: 
    failed to create NodePool np-zsxxnl: failed waiting for nodepool="np-zsxxnl" for cluster "cluster-candidate-4-22-zsxxnl" in resourcegroup="rg-candidate-4-22-7822rkz4qt8d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fd0879a-ebbb-4fea-b214-ece8494edab0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9fd0879a-ebbb-4fea-b214-ece8494edab0",
      "name": "9fd0879a-ebbb-4fea-b214-ece8494edab0",
      "status": "Failed",
      "startTime": "2026-10-03T03:22:25.37619508Z",
      "endTime": "2026-10-03T03:41:30.020299253Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-6hmr2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-b6wrl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-6hmr2: Creating: virtualmachine creating or updating\nMachine r5g9o6x4t5g2e7e-np-zsxxnl-wkz99-b6wrl: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x933c5f12a60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lbkmsp45kpxz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efe85861-3de4-4231-94f4-c87043439307
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efe85861-3de4-4231-94f4-c87043439307",
      "name": "efe85861-3de4-4231-94f4-c87043439307",
      "status": "Failed",
      "startTime": "2026-10-03T03:54:52.260057343Z",
      "endTime": "2026-10-03T04:13:53.608229594Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-hcvpc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-mgh9q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-hcvpc: Creating: virtualmachine creating or updating\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-mgh9q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x933c5f12a60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lbkmsp45kpxz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efe85861-3de4-4231-94f4-c87043439307
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/efe85861-3de4-4231-94f4-c87043439307",
      "name": "efe85861-3de4-4231-94f4-c87043439307",
      "status": "Failed",
      "startTime": "2026-10-03T03:54:52.260057343Z",
      "endTime": "2026-10-03T04:13:53.608229594Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-hcvpc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-mgh9q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-hcvpc: Creating: virtualmachine creating or updating\nMachine k7l8y6q4q2l9i3h-np-delete-main-68dll-mgh9q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xe7412ec40a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-j2962r94fd6x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4588d6e6-e963-44fd-8b18-89f83b2a899d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4588d6e6-e963-44fd-8b18-89f83b2a899d",
      "name": "4588d6e6-e963-44fd-8b18-89f83b2a899d",
      "status": "Failed",
      "startTime": "2026-10-03T03:00:19.581988952Z",
      "endTime": "2026-10-03T03:19:19.770602537Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xe7412ec40a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-j2962r94fd6x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4588d6e6-e963-44fd-8b18-89f83b2a899d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4588d6e6-e963-44fd-8b18-89f83b2a899d",
      "name": "4588d6e6-e963-44fd-8b18-89f83b2a899d",
      "status": "Failed",
      "startTime": "2026-10-03T03:00:19.581988952Z",
      "endTime": "2026-10-03T03:19:19.770602537Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x21c259b32dc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2sgbstf7nlxj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3898db04-618a-4491-92e8-a44483e69093
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3898db04-618a-4491-92e8-a44483e69093",
      "name": "3898db04-618a-4491-92e8-a44483e69093",
      "status": "Failed",
      "startTime": "2026-10-03T02:58:32.753298506Z",
      "endTime": "2026-10-03T03:17:39.788365749Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5r0r0n9t0l1p2d-np-1-dr752-hbt2g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m5r0r0n9t0l1p2d-np-1-dr752-zpxs2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5r0r0n9t0l1p2d-np-1-dr752-hbt2g: Creating: virtualmachine creating or updating\nMachine m5r0r0n9t0l1p2d-np-1-dr752-zpxs2: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x21c259b32dc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2sgbstf7nlxj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3898db04-618a-4491-92e8-a44483e69093
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3898db04-618a-4491-92e8-a44483e69093",
      "name": "3898db04-618a-4491-92e8-a44483e69093",
      "status": "Failed",
      "startTime": "2026-10-03T02:58:32.753298506Z",
      "endTime": "2026-10-03T03:17:39.788365749Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5r0r0n9t0l1p2d-np-1-dr752-hbt2g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m5r0r0n9t0l1p2d-np-1-dr752-zpxs2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5r0r0n9t0l1p2d-np-1-dr752-hbt2g: Creating: virtualmachine creating or updating\nMachine m5r0r0n9t0l1p2d-np-1-dr752-zpxs2: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6tqvsp" for cluster "cluster-candidate-4-21-6tqvsp"
Unexpected error:
    <*fmt.wrapError | 0x15f1e13c9ee0>: 
    failed to create NodePool np-6tqvsp: failed waiting for nodepool="np-6tqvsp" for cluster "cluster-candidate-4-21-6tqvsp" in resourcegroup="rg-candidate-4-21-qxp54ksqdkvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd90e427-e90a-4449-9d9c-98919a3e2323
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd90e427-e90a-4449-9d9c-98919a3e2323",
      "name": "cd90e427-e90a-4449-9d9c-98919a3e2323",
      "status": "Failed",
      "startTime": "2026-10-03T03:23:37.870437581Z",
      "endTime": "2026-10-03T03:42:43.397028044Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qxp54ksqdkvb-candidate-6tqvsp--managed/providers/Microsoft.Compute/virtualMachines/y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5 to exist\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-cm8m4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-cm8m4: Creating: virtualmachine creating or updating\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qxp54ksqdkvb-candidate-6tqvsp--managed/providers/Microsoft.Compute/virtualMachines/y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6tqvsp" for cluster "cluster-candidate-4-21-6tqvsp"
Unexpected error:
    <*fmt.wrapError | 0x15f1e13c9ee0>: 
    failed to create NodePool np-6tqvsp: failed waiting for nodepool="np-6tqvsp" for cluster "cluster-candidate-4-21-6tqvsp" in resourcegroup="rg-candidate-4-21-qxp54ksqdkvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd90e427-e90a-4449-9d9c-98919a3e2323
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd90e427-e90a-4449-9d9c-98919a3e2323",
      "name": "cd90e427-e90a-4449-9d9c-98919a3e2323",
      "status": "Failed",
      "startTime": "2026-10-03T03:23:37.870437581Z",
      "endTime": "2026-10-03T03:42:43.397028044Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qxp54ksqdkvb-candidate-6tqvsp--managed/providers/Microsoft.Compute/virtualMachines/y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5 to exist\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-cm8m4: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-cm8m4: Creating: virtualmachine creating or updating\nMachine y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qxp54ksqdkvb-candidate-6tqvsp--managed/providers/Microsoft.Compute/virtualMachines/y1i0m6w3c6r9w2i-np-6tqvsp-mg8xh-88lr5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vlg9bq" for cluster "cluster-candidate-4-23-vlg9bq"
Unexpected error:
    <*fmt.wrapError | 0x1e6e62f6cda0>: 
    failed to create NodePool np-vlg9bq: failed waiting for nodepool="np-vlg9bq" for cluster "cluster-candidate-4-23-vlg9bq" in resourcegroup="rg-candidate-4-23-dfclsfklgxtx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98169a27-755e-48b1-800f-32bc5e94d96c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98169a27-755e-48b1-800f-32bc5e94d96c",
      "name": "98169a27-755e-48b1-800f-32bc5e94d96c",
      "status": "Failed",
      "startTime": "2026-10-03T03:22:35.181930151Z",
      "endTime": "2026-10-03T03:41:39.859849458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-mqggg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-zfwnx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-mqggg: Creating: virtualmachine creating or updating\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-zfwnx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vlg9bq" for cluster "cluster-candidate-4-23-vlg9bq"
Unexpected error:
    <*fmt.wrapError | 0x1e6e62f6cda0>: 
    failed to create NodePool np-vlg9bq: failed waiting for nodepool="np-vlg9bq" for cluster "cluster-candidate-4-23-vlg9bq" in resourcegroup="rg-candidate-4-23-dfclsfklgxtx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98169a27-755e-48b1-800f-32bc5e94d96c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98169a27-755e-48b1-800f-32bc5e94d96c",
      "name": "98169a27-755e-48b1-800f-32bc5e94d96c",
      "status": "Failed",
      "startTime": "2026-10-03T03:22:35.181930151Z",
      "endTime": "2026-10-03T03:41:39.859849458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-mqggg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-zfwnx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-mqggg: Creating: virtualmachine creating or updating\nMachine g4t5b9n2y6n5g3x-np-vlg9bq-nplgj-zfwnx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7k8q7d" for cluster "cluster-candidate-4-20-7k8q7d"
Unexpected error:
    <*fmt.wrapError | 0x342eb43e6c40>: 
    failed to create NodePool np-7k8q7d: failed waiting for nodepool="np-7k8q7d" for cluster "cluster-candidate-4-20-7k8q7d" in resourcegroup="rg-candidate-4-20-vn2gzb9bgx68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f5da487-7370-42cc-8931-634a89f3149c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f5da487-7370-42cc-8931-634a89f3149c",
      "name": "8f5da487-7370-42cc-8931-634a89f3149c",
      "status": "Failed",
      "startTime": "2026-10-03T03:18:20.263677419Z",
      "endTime": "2026-10-03T03:37:30.013101747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-rlpsx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-v5gzq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-rlpsx: Creating: virtualmachine creating or updating\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-v5gzq: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7k8q7d" for cluster "cluster-candidate-4-20-7k8q7d"
Unexpected error:
    <*fmt.wrapError | 0x342eb43e6c40>: 
    failed to create NodePool np-7k8q7d: failed waiting for nodepool="np-7k8q7d" for cluster "cluster-candidate-4-20-7k8q7d" in resourcegroup="rg-candidate-4-20-vn2gzb9bgx68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f5da487-7370-42cc-8931-634a89f3149c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f5da487-7370-42cc-8931-634a89f3149c",
      "name": "8f5da487-7370-42cc-8931-634a89f3149c",
      "status": "Failed",
      "startTime": "2026-10-03T03:18:20.263677419Z",
      "endTime": "2026-10-03T03:37:30.013101747Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-rlpsx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-v5gzq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-rlpsx: Creating: virtualmachine creating or updating\nMachine n2s3g3y6j4i7c5t-np-7k8q7d-vrbbm-v5gzq: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xca0cd836f80>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-xrjvzm8fxrgt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e0749eb-04f8-4fdd-b9be-8393584f3697
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e0749eb-04f8-4fdd-b9be-8393584f3697",
      "name": "4e0749eb-04f8-4fdd-b9be-8393584f3697",
      "status": "Failed",
      "startTime": "2026-10-03T01:51:49.926221366Z",
      "endTime": "2026-10-03T02:10:56.26372719Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-45vn5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-bswmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-45vn5: Creating: virtualmachine creating or updating\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-bswmx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xca0cd836f80>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-xrjvzm8fxrgt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e0749eb-04f8-4fdd-b9be-8393584f3697
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e0749eb-04f8-4fdd-b9be-8393584f3697",
      "name": "4e0749eb-04f8-4fdd-b9be-8393584f3697",
      "status": "Failed",
      "startTime": "2026-10-03T01:51:49.926221366Z",
      "endTime": "2026-10-03T02:10:56.26372719Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-45vn5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-bswmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-45vn5: Creating: virtualmachine creating or updating\nMachine m9l6j7a1w6x8b0s-np-deletable-z98x4-bswmx: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x1363e14cc018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-jtmvgv9m6829" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31894849-27e9-4053-a67e-9e3babf2ad8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31894849-27e9-4053-a67e-9e3babf2ad8a",
      "name": "31894849-27e9-4053-a67e-9e3babf2ad8a",
      "status": "Failed",
      "startTime": "2026-10-03T01:01:05.764058109Z",
      "endTime": "2026-10-03T01:20:12.852653102Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-wxgnc-9w4jt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-wxgnc-clcm7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-wxgnc-9w4jt: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-wxgnc-clcm7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x1363e14cc018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-jtmvgv9m6829" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31894849-27e9-4053-a67e-9e3babf2ad8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31894849-27e9-4053-a67e-9e3babf2ad8a",
      "name": "31894849-27e9-4053-a67e-9e3babf2ad8a",
      "status": "Failed",
      "startTime": "2026-10-03T01:01:05.764058109Z",
      "endTime": "2026-10-03T01:20:12.852653102Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-wxgnc-9w4jt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-wxgnc-clcm7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-wxgnc-9w4jt: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-wxgnc-clcm7: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7m65d4" for cluster "cluster-candidate-4-21-7m65d4"
Unexpected error:
    <*fmt.wrapError | 0xdd74a91040>: 
    failed to create NodePool np-7m65d4: failed waiting for nodepool="np-7m65d4" for cluster "cluster-candidate-4-21-7m65d4" in resourcegroup="rg-candidate-4-21-cdkldcpn7c4w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f45b9d58-6bb8-408f-943e-164ec6193300
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f45b9d58-6bb8-408f-943e-164ec6193300",
      "name": "f45b9d58-6bb8-408f-943e-164ec6193300",
      "status": "Failed",
      "startTime": "2026-10-03T01:23:06.848197915Z",
      "endTime": "2026-10-03T01:42:11.810063991Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-7jtvb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-8fd6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-7jtvb: Creating: virtualmachine creating or updating\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-8fd6d: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7m65d4" for cluster "cluster-candidate-4-21-7m65d4"
Unexpected error:
    <*fmt.wrapError | 0xdd74a91040>: 
    failed to create NodePool np-7m65d4: failed waiting for nodepool="np-7m65d4" for cluster "cluster-candidate-4-21-7m65d4" in resourcegroup="rg-candidate-4-21-cdkldcpn7c4w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f45b9d58-6bb8-408f-943e-164ec6193300
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f45b9d58-6bb8-408f-943e-164ec6193300",
      "name": "f45b9d58-6bb8-408f-943e-164ec6193300",
      "status": "Failed",
      "startTime": "2026-10-03T01:23:06.848197915Z",
      "endTime": "2026-10-03T01:42:11.810063991Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-7jtvb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-8fd6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-7jtvb: Creating: virtualmachine creating or updating\nMachine z8t8u8s2s4m5t9g-np-7m65d4-6msht-8fd6d: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x356eae117080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-2xqzzh8h7jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2692539-c900-4d9b-9952-7a76ad7fc6a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2692539-c900-4d9b-9952-7a76ad7fc6a0",
      "name": "c2692539-c900-4d9b-9952-7a76ad7fc6a0",
      "status": "Failed",
      "startTime": "2026-10-03T00:56:20.964200965Z",
      "endTime": "2026-10-03T01:15:22.786636188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-kbv8l-5qvk6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-kbv8l-5qvk6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x356eae117080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-2xqzzh8h7jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2692539-c900-4d9b-9952-7a76ad7fc6a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2692539-c900-4d9b-9952-7a76ad7fc6a0",
      "name": "c2692539-c900-4d9b-9952-7a76ad7fc6a0",
      "status": "Failed",
      "startTime": "2026-10-03T00:56:20.964200965Z",
      "endTime": "2026-10-03T01:15:22.786636188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-kbv8l-5qvk6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-kbv8l-5qvk6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x36891cc75140>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-rw4pgrx2qsqc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a719f074-9cbb-4f60-8f74-9acd323de195
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a719f074-9cbb-4f60-8f74-9acd323de195",
      "name": "a719f074-9cbb-4f60-8f74-9acd323de195",
      "status": "Failed",
      "startTime": "2026-10-03T00:58:50.620270926Z",
      "endTime": "2026-10-03T01:17:53.07846241Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-nmjht: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-qnrnm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-nmjht: Creating: virtualmachine creating or updating\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-qnrnm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x36891cc75140>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-rw4pgrx2qsqc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a719f074-9cbb-4f60-8f74-9acd323de195
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a719f074-9cbb-4f60-8f74-9acd323de195",
      "name": "a719f074-9cbb-4f60-8f74-9acd323de195",
      "status": "Failed",
      "startTime": "2026-10-03T00:58:50.620270926Z",
      "endTime": "2026-10-03T01:17:53.07846241Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-nmjht: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-qnrnm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-nmjht: Creating: virtualmachine creating or updating\nMachine y4c5l7p2o7h6i3y-np-1-7slbg-qnrnm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7bgsbj" for cluster "cluster-candidate-5-1-7bgsbj"
Unexpected error:
    <*fmt.wrapError | 0x355ed6679960>: 
    failed to create NodePool np-7bgsbj: failed waiting for nodepool="np-7bgsbj" for cluster "cluster-candidate-5-1-7bgsbj" in resourcegroup="rg-candidate-5-1-twzxzjn6gcp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/48ddacde-704d-413e-818f-fb753e5b6f29
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/48ddacde-704d-413e-818f-fb753e5b6f29",
      "name": "48ddacde-704d-413e-818f-fb753e5b6f29",
      "status": "Failed",
      "startTime": "2026-10-03T01:21:40.063888823Z",
      "endTime": "2026-10-03T01:40:41.803400223Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-b2m7f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-ssxpc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-b2m7f: Creating: virtualmachine creating or updating\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-ssxpc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-7bgsbj" for cluster "cluster-candidate-5-1-7bgsbj"
Unexpected error:
    <*fmt.wrapError | 0x355ed6679960>: 
    failed to create NodePool np-7bgsbj: failed waiting for nodepool="np-7bgsbj" for cluster "cluster-candidate-5-1-7bgsbj" in resourcegroup="rg-candidate-5-1-twzxzjn6gcp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/48ddacde-704d-413e-818f-fb753e5b6f29
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/48ddacde-704d-413e-818f-fb753e5b6f29",
      "name": "48ddacde-704d-413e-818f-fb753e5b6f29",
      "status": "Failed",
      "startTime": "2026-10-03T01:21:40.063888823Z",
      "endTime": "2026-10-03T01:40:41.803400223Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-b2m7f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-ssxpc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-b2m7f: Creating: virtualmachine creating or updating\nMachine h4l9i6c5a5m3n3t-np-7bgsbj-b5gdb-ssxpc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x382d24c28030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zvljfb77qpnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fd6903c4-bc0c-4634-b456-f0053f4e62ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fd6903c4-bc0c-4634-b456-f0053f4e62ce",
      "name": "fd6903c4-bc0c-4634-b456-f0053f4e62ce",
      "status": "Failed",
      "startTime": "2026-10-03T00:59:54.046743948Z",
      "endTime": "2026-10-03T01:19:03.00985984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-2hhfv-7r58z: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-2hhfv-fmwpb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-2hhfv-7r58z: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-2hhfv-fmwpb: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x382d24c28030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zvljfb77qpnf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fd6903c4-bc0c-4634-b456-f0053f4e62ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fd6903c4-bc0c-4634-b456-f0053f4e62ce",
      "name": "fd6903c4-bc0c-4634-b456-f0053f4e62ce",
      "status": "Failed",
      "startTime": "2026-10-03T00:59:54.046743948Z",
      "endTime": "2026-10-03T01:19:03.00985984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-2hhfv-7r58z: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-2hhfv-fmwpb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-2hhfv-7r58z: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-2hhfv-fmwpb: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mpbw9s" for cluster "cluster-candidate-5-0-mpbw9s"
Unexpected error:
    <*fmt.wrapError | 0xea20e6b7700>: 
    failed to create NodePool np-mpbw9s: failed waiting for nodepool="np-mpbw9s" for cluster "cluster-candidate-5-0-mpbw9s" in resourcegroup="rg-candidate-5-0-6l9pw288lpdv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49f8bd06-343a-4b31-8a4d-21de9398ca35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49f8bd06-343a-4b31-8a4d-21de9398ca35",
      "name": "49f8bd06-343a-4b31-8a4d-21de9398ca35",
      "status": "Failed",
      "startTime": "2026-10-03T01:23:58.4098212Z",
      "endTime": "2026-10-03T01:43:01.804835963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-sx8g2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-xvdjz: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-sx8g2: Creating: virtualmachine creating or updating\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-xvdjz: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-mpbw9s" for cluster "cluster-candidate-5-0-mpbw9s"
Unexpected error:
    <*fmt.wrapError | 0xea20e6b7700>: 
    failed to create NodePool np-mpbw9s: failed waiting for nodepool="np-mpbw9s" for cluster "cluster-candidate-5-0-mpbw9s" in resourcegroup="rg-candidate-5-0-6l9pw288lpdv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49f8bd06-343a-4b31-8a4d-21de9398ca35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49f8bd06-343a-4b31-8a4d-21de9398ca35",
      "name": "49f8bd06-343a-4b31-8a4d-21de9398ca35",
      "status": "Failed",
      "startTime": "2026-10-03T01:23:58.4098212Z",
      "endTime": "2026-10-03T01:43:01.804835963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-sx8g2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-xvdjz: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-sx8g2: Creating: virtualmachine creating or updating\nMachine k9g6t4i6h6q8k7w-np-mpbw9s-tmszd-xvdjz: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-55sh69" for cluster "cluster-candidate-4-23-55sh69"
Unexpected error:
    <*fmt.wrapError | 0x2d83ca910b00>: 
    failed to create NodePool np-55sh69: failed waiting for nodepool="np-55sh69" for cluster "cluster-candidate-4-23-55sh69" in resourcegroup="rg-candidate-4-23-b2jh9rr7848q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/857aaaff-55f2-4507-bff3-f5f99ad82196
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/857aaaff-55f2-4507-bff3-f5f99ad82196",
      "name": "857aaaff-55f2-4507-bff3-f5f99ad82196",
      "status": "Failed",
      "startTime": "2026-10-03T01:21:00.037991541Z",
      "endTime": "2026-10-03T01:40:02.219519171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-qt7pp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-xbwxp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-qt7pp: Creating: virtualmachine creating or updating\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-xbwxp: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-55sh69" for cluster "cluster-candidate-4-23-55sh69"
Unexpected error:
    <*fmt.wrapError | 0x2d83ca910b00>: 
    failed to create NodePool np-55sh69: failed waiting for nodepool="np-55sh69" for cluster "cluster-candidate-4-23-55sh69" in resourcegroup="rg-candidate-4-23-b2jh9rr7848q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/857aaaff-55f2-4507-bff3-f5f99ad82196
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/857aaaff-55f2-4507-bff3-f5f99ad82196",
      "name": "857aaaff-55f2-4507-bff3-f5f99ad82196",
      "status": "Failed",
      "startTime": "2026-10-03T01:21:00.037991541Z",
      "endTime": "2026-10-03T01:40:02.219519171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-qt7pp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-xbwxp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-qt7pp: Creating: virtualmachine creating or updating\nMachine z7q7z3g1g0w5l4w-np-55sh69-kmbcq-xbwxp: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xcs2zx" for cluster "cluster-candidate-4-20-xcs2zx"
Unexpected error:
    <*fmt.wrapError | 0x3bb65f4dc3e0>: 
    failed to create NodePool np-xcs2zx: failed waiting for nodepool="np-xcs2zx" for cluster "cluster-candidate-4-20-xcs2zx" in resourcegroup="rg-candidate-4-20-9tm8m64jwh6q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5e3d740-6442-4b8e-9013-9867a8b30bc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5e3d740-6442-4b8e-9013-9867a8b30bc0",
      "name": "b5e3d740-6442-4b8e-9013-9867a8b30bc0",
      "status": "Failed",
      "startTime": "2026-10-03T01:20:28.820955232Z",
      "endTime": "2026-10-03T01:39:31.794768159Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-42gm5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-zmvl5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-42gm5: Creating: virtualmachine creating or updating\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-zmvl5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xcs2zx" for cluster "cluster-candidate-4-20-xcs2zx"
Unexpected error:
    <*fmt.wrapError | 0x3bb65f4dc3e0>: 
    failed to create NodePool np-xcs2zx: failed waiting for nodepool="np-xcs2zx" for cluster "cluster-candidate-4-20-xcs2zx" in resourcegroup="rg-candidate-4-20-9tm8m64jwh6q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5e3d740-6442-4b8e-9013-9867a8b30bc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5e3d740-6442-4b8e-9013-9867a8b30bc0",
      "name": "b5e3d740-6442-4b8e-9013-9867a8b30bc0",
      "status": "Failed",
      "startTime": "2026-10-03T01:20:28.820955232Z",
      "endTime": "2026-10-03T01:39:31.794768159Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-42gm5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-zmvl5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-42gm5: Creating: virtualmachine creating or updating\nMachine b5o5a6z5q4p5e0b-np-xcs2zx-hlrdg-zmvl5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-md77w4" for cluster "cluster-candidate-4-22-md77w4"
Unexpected error:
    <*fmt.wrapError | 0x34d9aee24b60>: 
    failed to create NodePool np-md77w4: failed waiting for nodepool="np-md77w4" for cluster "cluster-candidate-4-22-md77w4" in resourcegroup="rg-candidate-4-22-qt94z5jzjgsz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7b70034-c32d-4689-8c16-d6b19ed89cc8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7b70034-c32d-4689-8c16-d6b19ed89cc8",
      "name": "c7b70034-c32d-4689-8c16-d6b19ed89cc8",
      "status": "Failed",
      "startTime": "2026-10-03T01:25:32.783100892Z",
      "endTime": "2026-10-03T01:44:41.785015844Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-488zq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-f2khf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-488zq: Creating: virtualmachine creating or updating\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-f2khf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-md77w4" for cluster "cluster-candidate-4-22-md77w4"
Unexpected error:
    <*fmt.wrapError | 0x34d9aee24b60>: 
    failed to create NodePool np-md77w4: failed waiting for nodepool="np-md77w4" for cluster "cluster-candidate-4-22-md77w4" in resourcegroup="rg-candidate-4-22-qt94z5jzjgsz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7b70034-c32d-4689-8c16-d6b19ed89cc8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c7b70034-c32d-4689-8c16-d6b19ed89cc8",
      "name": "c7b70034-c32d-4689-8c16-d6b19ed89cc8",
      "status": "Failed",
      "startTime": "2026-10-03T01:25:32.783100892Z",
      "endTime": "2026-10-03T01:44:41.785015844Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-488zq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-f2khf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-488zq: Creating: virtualmachine creating or updating\nMachine m6p3c2b3z7i0g2t-np-md77w4-k4nbk-f2khf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xbbf668ab320>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-lqwbkg" in resourcegroup="rg-np-version-upgrade-lqwbkg-jl7ktvcrp5z8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4143c12-1ac1-4d68-ba18-1bb777cf81b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4143c12-1ac1-4d68-ba18-1bb777cf81b4",
      "name": "a4143c12-1ac1-4d68-ba18-1bb777cf81b4",
      "status": "Failed",
      "startTime": "2026-10-03T01:58:23.770970582Z",
      "endTime": "2026-10-03T02:17:26.319658092Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lqwbkg-jl7ktvcrp5z8-np-upgrade-lq-e094d963/providers/Microsoft.Compute/virtualMachines/q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lqwbkg-jl7ktvcrp5z8-np-upgrade-lq-e094d963/providers/Microsoft.Compute/virtualMachines/q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xbbf668ab320>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-lqwbkg" in resourcegroup="rg-np-version-upgrade-lqwbkg-jl7ktvcrp5z8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4143c12-1ac1-4d68-ba18-1bb777cf81b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a4143c12-1ac1-4d68-ba18-1bb777cf81b4",
      "name": "a4143c12-1ac1-4d68-ba18-1bb777cf81b4",
      "status": "Failed",
      "startTime": "2026-10-03T01:58:23.770970582Z",
      "endTime": "2026-10-03T02:17:26.319658092Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lqwbkg-jl7ktvcrp5z8-np-upgrade-lq-e094d963/providers/Microsoft.Compute/virtualMachines/q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lqwbkg-jl7ktvcrp5z8-np-upgrade-lq-e094d963/providers/Microsoft.Compute/virtualMachines/q0g7j2k6j1j2c5g-npupgrade-4-21-z7zbc-cgvzh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred