alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T00:40:12Z
  Ended:       2026-10-01T00:54:14Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7664896-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003c2f500>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4t7wr4mxhmz6" to finish creating: GET https://rp.j2153728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "name": "911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "status": "Failed",
      "startTime": "2026-09-30T23:49:39.505681667Z",
      "endTime": "2026-10-01T00:08:45.705788813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003c2f500>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4t7wr4mxhmz6" to finish creating: GET https://rp.j2153728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "name": "911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "status": "Failed",
      "startTime": "2026-09-30T23:49:39.505681667Z",
      "endTime": "2026-10-01T00:08:45.705788813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 27 time(s) (20 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-9p7qd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-9p7qd has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-75htw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-75htw has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-hsnlr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-hsnlr has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-bdq7h", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-bdq7h has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-t7t2f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-t7t2f has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-c472z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-c472z has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-6j8pl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-6j8pl has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-5vmb4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-5vmb4 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-z5vtc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-z5vtc has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-9llk2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-9llk2 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-b79df", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-b79df has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-pgcdv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-pgcdv has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-cm978", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-cm978 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-bb46w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-bb46w has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-sqbrn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-sqbrn has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-kmksg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-kmksg has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-s5k4b", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-s5k4b has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-v55n5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-v55n5 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-lxq4s", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-lxq4s has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-xfjzk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-xfjzk has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:287]: failed to scale down node pool "np-scale" from 2 to 1 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc00217d260>: 
    failed waiting for nodepool="np-scale" in cluster="priv-np-scale" resourcegroup="priv-np-scale-d7bh2pbbtldm" to finish updating, caused by: timeout '60.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-scale in cluster priv-np-scale in resource group priv-np-scale-d7bh2pbbtldm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:287]: failed to scale down node pool "np-scale" from 2 to 1 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc00217d260>: 
    failed waiting for nodepool="np-scale" in cluster="priv-np-scale" resourcegroup="priv-np-scale-d7bh2pbbtldm" to finish updating, caused by: timeout '60.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-scale in cluster priv-np-scale in resource group priv-np-scale-d7bh2pbbtldm, error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T23:47:21Z
  Ended:       2026-09-30T23:53:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2035712-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5ki3fusaq8plth807pisqonnpgbdn6-e4t7z1r7k6m9l4f", pod="kube-state-metrics-hcp-754f855b69-774l4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5ki3fusaq8plth807pisqonnpgbdn6-e4t7z1r7k6m9l4f/kube-state-metrics-hcp-754f855b69-774l4 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T23:20:44Z
  Ended:       2026-09-30T23:26:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4901760-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5k6opsebogjptm5uab2qhhik4rqbt7-k4w6n5p5s8z3z6k", pod="azure-file-csi-driver-controller-7c869d77d7-z84jf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5k6opsebogjptm5uab2qhhik4rqbt7-k4w6n5p5s8z3z6k/azure-file-csi-driver-controller-7c869d77d7-z84jf has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b7a960>: 
    failed to create HCP cluster ea-listnpdfsm: failed waiting for cluster="ea-listnpdfsm" in resourcegroup="ea-list-g57slgdp8bj6" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b7a960>: 
    failed to create HCP cluster ea-listnpdfsm: failed waiting for cluster="ea-listnpdfsm" in resourcegroup="ea-list-g57slgdp8bj6" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000cda1a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s9lfl9dtlsjp" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000cda1a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s9lfl9dtlsjp" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-9xv2vk78f5m7" to finish creating: GET https://rp.j4339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "name": "9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "status": "Failed",
  "startTime": "2026-09-30T22:15:27.259108656Z",
  "endTime": "2026-09-30T22:34:33.885780975Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-9xv2vk78f5m7" to finish creating: GET https://rp.j4339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "name": "9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "status": "Failed",
  "startTime": "2026-09-30T22:15:27.259108656Z",
  "endTime": "2026-09-30T22:34:33.885780975Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

time=2026-09-30T20:23:20.071Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T20:23:20.785Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j8920064-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j8920064-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T20:23:20.785Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:23:21.014Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T20:23:21.176Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T20:23:21.176Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:23:21.232Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:33:24.255Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:33:24.255Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:34:24.257Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T20:34:24.848Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j8920064-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j8920064-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T20:34:24.848Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:25.188Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T20:34:25.588Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:25.588Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:25.757Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T20:34:26.389Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:26.390Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:26.616Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:26.899Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T20:34:27.340Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T20:34:27.673Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:27.873Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:28.054Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:28.402Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T20:34:28.402Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:28.487Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T20:44:30.230Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:44:30.230Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:45:30.230Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T20:45:30.710Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j8920064-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j8920064-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T20:45:30.711Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:31.093Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T20:45:31.467Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:31.467Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:31.591Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T20:45:32.324Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:32.324Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:32.528Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:32.928Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T20:45:33.424Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T20:45:33.656Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:34.121Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:34.518Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:34.878Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T20:45:34.878Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:34.950Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T20:53:20.192Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T20:53:20.193Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:39:32Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j8920064-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j8920064-svc has been unreachable for 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:39:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8920064/providers/microsoft.monitor/accounts/services-j8920064", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-6rhvw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:40:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8920064/providers/microsoft.monitor/accounts/services-j8920064", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-6rhvw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-ptxgs", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-ptxgs has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-rfxj6", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-rfxj6 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:33:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-fwc59", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-fwc59 has been in a non-ready state for longer than 5 minutes.

time=2026-09-30T20:31:01.398Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-09-30T20:31:01.965Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-09-30T20:31:02.790Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=0a6bcb1a4c8db820be60ebb3431d76a20397d50a3cee281e89a5518a17e2b3e0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5173888-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F0a6bcb1a4c8db820be60ebb3431d76a20397d50a3cee281e89a5518a17e2b3e0
time=2026-09-30T20:33:03.396Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Resources/deployments/0a6bcb1a4c8db820be60ebb3431d76a20397d50a3cee281e89a5518a17e2b3e0/operationStatuses/08584108066230621918\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Resources/deployments/profile-mlyktaqazs6u6\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j5173888-mgmt-2/resourceAssociations/nsp-j5173888-mgmt-2-ra-30be3175-d83b-506a-923e-30e4da745648\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"TaskCanceledException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j5173888-mgmt-2/resourceAssociations/nsp-j5173888-mgmt-2-ra-30be3175-d83b-506a-923e-30e4da745648, Error message: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:175]: cluster verification failed
Unexpected error:
    <*errors.joinError | 0xc0015b4cc0>: 
    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: no VMs found for nodepool des-np in managed resource group des-encrypt-gngwklk9wp6n--managed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:175]: cluster verification failed
Unexpected error:
    <*errors.joinError | 0xc0015b4cc0>: 
    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: no VMs found for nodepool des-np in managed resource group des-encrypt-gngwklk9wp6n--managed
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:59:11Z
  Ended:       2026-09-30T20:14:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5203456-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p", pod="control-plane-operator-d4ff858b6-x6hb9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p/control-plane-operator-d4ff858b6-x6hb9 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:59:12Z
  Ended:       2026-09-30T20:14:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5203456-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p", pod="control-plane-operator-d4ff858b6-l2s48", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p/control-plane-operator-d4ff858b6-l2s48 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T19:59:12Z
  Ended:       2026-09-30T20:14:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5203456-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p", pod="control-plane-operator-d4ff858b6-4pbr7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p/control-plane-operator-d4ff858b6-4pbr7 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T19:59:12Z
  Ended:       2026-09-30T20:14:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5203456-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p", pod="control-plane-operator-d4ff858b6-wwpwh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p/control-plane-operator-d4ff858b6-wwpwh has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T20:00:11Z
  Ended:       2026-09-30T20:14:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5203456-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p", pod="control-plane-operator-d4ff858b6-tvjkl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5hgejihc22takh2irlnl38m4fd005m-q6p1l2n2a3b6z5p/control-plane-operator-d4ff858b6-tvjkl has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001666348>: 
    resource group "rg-candidate-5-1-7k2hspfgwlgc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jdl45b" in resourcegroup="rg-candidate-5-1-7k2hspfgwlgc" to finish deleting: GET https://rp.j2434176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048",
      "name": "77026e76-9732-4942-8d30-76c6571b6048",
      "status": "Failed",
      "startTime": "2026-09-30T19:19:44.434042017Z",
      "endTime": "2026-09-30T19:43:54.082755547Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5gr9ar4ha50b9n6jmkh1qfntpjdb09 still exists (deletion dispatched at 2026-09-30T19:19:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001666348>: 
    resource group "rg-candidate-5-1-7k2hspfgwlgc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jdl45b" in resourcegroup="rg-candidate-5-1-7k2hspfgwlgc" to finish deleting: GET https://rp.j2434176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048",
      "name": "77026e76-9732-4942-8d30-76c6571b6048",
      "status": "Failed",
      "startTime": "2026-09-30T19:19:44.434042017Z",
      "endTime": "2026-09-30T19:43:54.082755547Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5gr9ar4ha50b9n6jmkh1qfntpjdb09 still exists (deletion dispatched at 2026-09-30T19:19:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-30T18:26:03.649Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T18:26:04.095Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0633472-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j0633472-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:42f17d2a-2d54-4fa2-b94e-06bedcfbde0d region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#42f17d2a-2d54-4fa2-b94e-06bedcfbde0d INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T18:26:04.095Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:26:04.389Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T18:26:04.602Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T18:26:04.602Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:26:04.633Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:36:07.312Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:36:07.312Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:37:07.314Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T18:37:08.026Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0633472-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j0633472-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:42f17d2a-2d54-4fa2-b94e-06bedcfbde0d region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#42f17d2a-2d54-4fa2-b94e-06bedcfbde0d INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T18:37:08.026Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:08.359Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T18:37:08.708Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:08.708Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:08.746Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T18:37:09.698Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:09.698Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:09.874Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:10.131Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T18:37:10.610Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T18:37:10.790Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:11.050Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:11.400Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:11.717Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T18:37:11.717Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:11.766Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T18:47:13.248Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:47:13.248Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:48:13.252Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T18:48:13.967Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0633472-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j0633472-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:42f17d2a-2d54-4fa2-b94e-06bedcfbde0d region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#42f17d2a-2d54-4fa2-b94e-06bedcfbde0d INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T18:48:13.968Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:14.374Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T18:48:14.654Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:14.654Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:14.778Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T18:48:15.694Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:15.694Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:16.096Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:16.376Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T18:48:16.694Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T18:48:16.931Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:17.265Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:17.464Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:17.811Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T18:48:17.811Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:17.876Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T18:56:03.654Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T18:56:03.655Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:42:29Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j0633472-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j0633472-svc has been unreachable for 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:42:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0633472/providers/microsoft.monitor/accounts/services-j0633472", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-kq5jf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:42:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0633472/providers/microsoft.monitor/accounts/services-j0633472", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-kq5jf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-qp5bd", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-qp5bd has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-x78jx", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-x78jx has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-55p2d", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-55p2d has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014f6680>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mxcczjbklf6k" to finish creating: GET https://rp.j1701248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a",
      "name": "49794eca-872f-4949-8230-a9c6d512647a",
      "status": "Failed",
      "startTime": "2026-09-30T19:06:11.188459377Z",
      "endTime": "2026-09-30T19:25:16.297661458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014f6680>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mxcczjbklf6k" to finish creating: GET https://rp.j1701248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a",
      "name": "49794eca-872f-4949-8230-a9c6d512647a",
      "status": "Failed",
      "startTime": "2026-09-30T19:06:11.188459377Z",
      "endTime": "2026-09-30T19:25:16.297661458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000e5b548>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000e5b548>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001128760>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-h59xkph9dfpm" to finish creating: GET https://rp.j5347072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c",
      "name": "88da31a3-069e-4240-80a2-676685212f2c",
      "status": "Failed",
      "startTime": "2026-09-30T18:49:41.355729108Z",
      "endTime": "2026-09-30T19:18:42.930044888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001128760>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-h59xkph9dfpm" to finish creating: GET https://rp.j5347072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c",
      "name": "88da31a3-069e-4240-80a2-676685212f2c",
      "status": "Failed",
      "startTime": "2026-09-30T18:49:41.355729108Z",
      "endTime": "2026-09-30T19:18:42.930044888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:14:32Z
  Ended:       2026-09-30T19:40:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="009dd713-cc58-4cfb-b82f-f1fca7b413d6"
  Description: Pod ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:14:32Z
  Ended:       2026-09-30T19:41:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="009dd713-cc58-4cfb-b82f-f1fca7b413d6"
  Description: Pod ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:11:34Z
  Ended:       2026-09-30T19:36:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:22:36Z
  Ended:       2026-09-30T19:36:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:25:32Z
  Ended:       2026-09-30T19:38:40Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:25:32Z
  Ended:       2026-09-30T19:38:40Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00120c640>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mnp9svrwqcpp" to finish creating: GET https://rp.j4426752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "name": "9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "status": "Failed",
      "startTime": "2026-09-30T18:21:17.653151468Z",
      "endTime": "2026-09-30T18:50:22.965863416Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00120c640>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mnp9svrwqcpp" to finish creating: GET https://rp.j4426752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "name": "9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "status": "Failed",
      "startTime": "2026-09-30T18:21:17.653151468Z",
      "endTime": "2026-09-30T18:50:22.965863416Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:226]: Timed out after 300.001s.
KAS→kubelet log path never became reachable from VM inside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:220 with:
no running pods found in openshift-kube-apiserver
Expected
    <string>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:226]: Timed out after 300.001s.
KAS→kubelet log path never became reachable from VM inside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:220 with:
no running pods found in openshift-kube-apiserver
Expected
    <string>: ...
not to be empty

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T18:52:59Z
  Ended:       2026-09-30T19:08:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T18:52:59Z
  Ended:       2026-09-30T19:08:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T18:43:56Z
  Ended:       2026-09-30T19:12:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bce6bb08-1647-4613-853e-b0a5759423ff"
  Description: Pod ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-30T18:44:57Z
  Ended:       2026-09-30T19:12:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bce6bb08-1647-4613-853e-b0a5759423ff"
  Description: Pod ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T18:57:01Z
  Ended:       2026-09-30T19:10:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T18:57:01Z
  Ended:       2026-09-30T19:10:01Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:57:46Z
  Ended:       2026-09-30T18:21:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6280960-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5fn1t8q899f67e9c3gfapqs0nka9q4-z9j3x8s3e4f8g8i", pod="azure-file-csi-driver-operator-6fc68567db-xb6kc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5fn1t8q899f67e9c3gfapqs0nka9q4-z9j3x8s3e4f8g8i/azure-file-csi-driver-operator-6fc68567db-xb6kc has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:57:46Z
  Ended:       2026-09-30T18:21:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6280960-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5fn1t8q899f67e9c3gfapqs0nka9q4-z9j3x8s3e4f8g8i", pod="azure-file-csi-driver-operator-6fc68567db-ck5hv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5fn1t8q899f67e9c3gfapqs0nka9q4-z9j3x8s3e4f8g8i/azure-file-csi-driver-operator-6fc68567db-ck5hv has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:53:32Z
  Ended:       2026-09-30T18:09:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3795328-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:56:30Z
  Ended:       2026-09-30T18:09:30Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3795328-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:59:28Z
  Ended:       2026-09-30T18:09:30Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3795328-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-z8sw852rvcl6/cluster-candidate-5-1-f8bld9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f24640>: 
    failed to create HCP cluster cluster-candidate-5-1-f8bld9: failed waiting for cluster="cluster-candidate-5-1-f8bld9" in resourcegroup="rg-candidate-5-1-z8sw852rvcl6" to finish creating: GET https://rp.j5298560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736",
      "name": "1da1b003-1c66-47c8-99b2-f49235ea2736",
      "status": "Failed",
      "startTime": "2026-09-30T17:01:56.141378012Z",
      "endTime": "2026-09-30T17:30:57.762149233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 24m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-z8sw852rvcl6/cluster-candidate-5-1-f8bld9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f24640>: 
    failed to create HCP cluster cluster-candidate-5-1-f8bld9: failed waiting for cluster="cluster-candidate-5-1-f8bld9" in resourcegroup="rg-candidate-5-1-z8sw852rvcl6" to finish creating: GET https://rp.j5298560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736",
      "name": "1da1b003-1c66-47c8-99b2-f49235ea2736",
      "status": "Failed",
      "startTime": "2026-09-30T17:01:56.141378012Z",
      "endTime": "2026-09-30T17:30:57.762149233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 24m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 13 time(s) (1 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:13:21Z
  Ended:       2026-09-30T17:39:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5298560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f514d5mgc9haie801lsop9ne436rb-q0n1v0q8g3o5z5q", pod="router-8445c79445-9rnr2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f514d5mgc9haie801lsop9ne436rb-q0n1v0q8g3o5z5q/router-8445c79445-9rnr2 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00104c300>: 
    resource group "nodepool-delete-c4hb66lp5frs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c4hb66lp5frs" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e6cc5ae-d28a-4989-ad49-e0dd769288e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e6cc5ae-d28a-4989-ad49-e0dd769288e5",
      "name": "7e6cc5ae-d28a-4989-ad49-e0dd769288e5",
      "status": "Failed",
      "startTime": "2026-09-30T17:25:05.3374218Z",
      "endTime": "2026-09-30T17:49:05.822251371Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f18hpo95treletippqdt0q6nd6217 still exists (deletion dispatched at 2026-09-30T17:25:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00104c300>: 
    resource group "nodepool-delete-c4hb66lp5frs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c4hb66lp5frs" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e6cc5ae-d28a-4989-ad49-e0dd769288e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e6cc5ae-d28a-4989-ad49-e0dd769288e5",
      "name": "7e6cc5ae-d28a-4989-ad49-e0dd769288e5",
      "status": "Failed",
      "startTime": "2026-09-30T17:25:05.3374218Z",
      "endTime": "2026-09-30T17:49:05.822251371Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f18hpo95treletippqdt0q6nd6217 still exists (deletion dispatched at 2026-09-30T17:25:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00171ee58>: 
    resource group "v20261001preview-8xzpnp7tdgz2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-8xzpnp7tdgz2" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b4e803e-d3df-4fa2-8617-7c767d9fdb48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b4e803e-d3df-4fa2-8617-7c767d9fdb48",
      "name": "7b4e803e-d3df-4fa2-8617-7c767d9fdb48",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:45.793371573Z",
      "endTime": "2026-09-30T17:44:45.916509607Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0lc5egjrilphmb608mchbeokkhli still exists (deletion dispatched at 2026-09-30T17:20:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00171ee58>: 
    resource group "v20261001preview-8xzpnp7tdgz2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-8xzpnp7tdgz2" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b4e803e-d3df-4fa2-8617-7c767d9fdb48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b4e803e-d3df-4fa2-8617-7c767d9fdb48",
      "name": "7b4e803e-d3df-4fa2-8617-7c767d9fdb48",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:45.793371573Z",
      "endTime": "2026-09-30T17:44:45.916509607Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0lc5egjrilphmb608mchbeokkhli still exists (deletion dispatched at 2026-09-30T17:20:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000af62c0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fs49jjvbxrlk" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "name": "e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "status": "Failed",
      "startTime": "2026-09-30T16:53:07.232391508Z",
      "endTime": "2026-09-30T17:22:15.941466136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m42s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000af62c0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fs49jjvbxrlk" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "name": "e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "status": "Failed",
      "startTime": "2026-09-30T16:53:07.232391508Z",
      "endTime": "2026-09-30T17:22:15.941466136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m42s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002518090>: 
    resource group "node-limit-test-2jbt488tkpl7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2jbt488tkpl7" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "name": "da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "status": "Failed",
      "startTime": "2026-09-30T17:02:22.062008253Z",
      "endTime": "2026-09-30T17:26:25.935485649Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1mqu517if66tcq7vv1883rm6svr2 still exists (deletion dispatched at 2026-09-30T17:02:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002518090>: 
    resource group "node-limit-test-2jbt488tkpl7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2jbt488tkpl7" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "name": "da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "status": "Failed",
      "startTime": "2026-09-30T17:02:22.062008253Z",
      "endTime": "2026-09-30T17:26:25.935485649Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1mqu517if66tcq7vv1883rm6svr2 still exists (deletion dispatched at 2026-09-30T17:02:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001b32210>: 
    resource group "rg-candidate-4-21-fjd2gblvhgpk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-jb8n5l" in resourcegroup="rg-candidate-4-21-fjd2gblvhgpk" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cbfc4ac-c696-45d2-b553-5a272022650d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cbfc4ac-c696-45d2-b553-5a272022650d",
      "name": "9cbfc4ac-c696-45d2-b553-5a272022650d",
      "status": "Failed",
      "startTime": "2026-09-30T17:14:14.963393723Z",
      "endTime": "2026-09-30T17:38:16.020897196Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0e8dbaqbo8t5dt5vdcasi8k08a4q still exists (deletion dispatched at 2026-09-30T17:14:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001b32210>: 
    resource group "rg-candidate-4-21-fjd2gblvhgpk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-jb8n5l" in resourcegroup="rg-candidate-4-21-fjd2gblvhgpk" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cbfc4ac-c696-45d2-b553-5a272022650d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cbfc4ac-c696-45d2-b553-5a272022650d",
      "name": "9cbfc4ac-c696-45d2-b553-5a272022650d",
      "status": "Failed",
      "startTime": "2026-09-30T17:14:14.963393723Z",
      "endTime": "2026-09-30T17:38:16.020897196Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0e8dbaqbo8t5dt5vdcasi8k08a4q still exists (deletion dispatched at 2026-09-30T17:14:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00280c2a0>: 
    resource group "rg-candidate-5-0-ns2cs4dsdbxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-spnhk7" in resourcegroup="rg-candidate-5-0-ns2cs4dsdbxw" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7972329-b9a1-4d78-b6ee-4512a34c9cf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7972329-b9a1-4d78-b6ee-4512a34c9cf8",
      "name": "c7972329-b9a1-4d78-b6ee-4512a34c9cf8",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:04.153634774Z",
      "endTime": "2026-09-30T17:44:05.883180823Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0ebuj103h4dn4jek0oofbev66a3k still exists (deletion dispatched at 2026-09-30T17:20:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00280c2a0>: 
    resource group "rg-candidate-5-0-ns2cs4dsdbxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-spnhk7" in resourcegroup="rg-candidate-5-0-ns2cs4dsdbxw" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7972329-b9a1-4d78-b6ee-4512a34c9cf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c7972329-b9a1-4d78-b6ee-4512a34c9cf8",
      "name": "c7972329-b9a1-4d78-b6ee-4512a34c9cf8",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:04.153634774Z",
      "endTime": "2026-09-30T17:44:05.883180823Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0ebuj103h4dn4jek0oofbev66a3k still exists (deletion dispatched at 2026-09-30T17:20:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-9z27ks86nc27/cluster-candidate-5-1-hrwtgg should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010a04c0>: 
    failed to create HCP cluster cluster-candidate-5-1-hrwtgg: failed waiting for cluster="cluster-candidate-5-1-hrwtgg" in resourcegroup="rg-candidate-5-1-9z27ks86nc27" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a",
      "name": "9135939f-e992-427f-be1e-4ecfae2b319a",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:29.681806994Z",
      "endTime": "2026-09-30T17:21:35.77427567Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 22m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-9z27ks86nc27/cluster-candidate-5-1-hrwtgg should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010a04c0>: 
    failed to create HCP cluster cluster-candidate-5-1-hrwtgg: failed waiting for cluster="cluster-candidate-5-1-hrwtgg" in resourcegroup="rg-candidate-5-1-9z27ks86nc27" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a",
      "name": "9135939f-e992-427f-be1e-4ecfae2b319a",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:29.681806994Z",
      "endTime": "2026-09-30T17:21:35.77427567Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 22m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00113a0c0>: 
    resource group "private-kas-hm9pdnmdszt5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-hm9pdnmdszt5" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf",
      "name": "2f588ad3-c1af-45bb-9548-096867a64bbf",
      "status": "Failed",
      "startTime": "2026-09-30T17:29:27.633769413Z",
      "endTime": "2026-09-30T17:53:32.876325702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0n6rovpqtd5op2d9vogdgft7tah1 still exists (deletion dispatched at 2026-09-30T17:29:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00113a0c0>: 
    resource group "private-kas-hm9pdnmdszt5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-hm9pdnmdszt5" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf",
      "name": "2f588ad3-c1af-45bb-9548-096867a64bbf",
      "status": "Failed",
      "startTime": "2026-09-30T17:29:27.633769413Z",
      "endTime": "2026-09-30T17:53:32.876325702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0n6rovpqtd5op2d9vogdgft7tah1 still exists (deletion dispatched at 2026-09-30T17:29:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-622nv7gb94zh/cluster-candidate-4-22-jcthkg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c422a0>: 
    failed to create HCP cluster cluster-candidate-4-22-jcthkg: failed waiting for cluster="cluster-candidate-4-22-jcthkg" in resourcegroup="rg-candidate-4-22-622nv7gb94zh" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "name": "6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:38.429049015Z",
      "endTime": "2026-09-30T17:21:45.984146934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 22m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-622nv7gb94zh/cluster-candidate-4-22-jcthkg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c422a0>: 
    failed to create HCP cluster cluster-candidate-4-22-jcthkg: failed waiting for cluster="cluster-candidate-4-22-jcthkg" in resourcegroup="rg-candidate-4-22-622nv7gb94zh" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "name": "6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:38.429049015Z",
      "endTime": "2026-09-30T17:21:45.984146934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 22m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0029f60f0>: 
    resource group "complex-cilium-kv-vkmqwlbstl86": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vkmqwlbstl86" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/758b0048-541b-41f7-b362-273a15532107
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/758b0048-541b-41f7-b362-273a15532107",
      "name": "758b0048-541b-41f7-b362-273a15532107",
      "status": "Failed",
      "startTime": "2026-09-30T17:24:38.640589117Z",
      "endTime": "2026-09-30T17:48:45.86210432Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1cult37rnvjnunif5m4d75jg68nc still exists (deletion dispatched at 2026-09-30T17:24:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0029f60f0>: 
    resource group "complex-cilium-kv-vkmqwlbstl86": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vkmqwlbstl86" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/758b0048-541b-41f7-b362-273a15532107
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/758b0048-541b-41f7-b362-273a15532107",
      "name": "758b0048-541b-41f7-b362-273a15532107",
      "status": "Failed",
      "startTime": "2026-09-30T17:24:38.640589117Z",
      "endTime": "2026-09-30T17:48:45.86210432Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1cult37rnvjnunif5m4d75jg68nc still exists (deletion dispatched at 2026-09-30T17:24:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010c0210>: 
    resource group "private-keyvault-9g9ksdvdr2w9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-9g9ksdvdr2w9" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2",
      "name": "135095ef-4130-48de-a890-36a3f2ec4ba2",
      "status": "Failed",
      "startTime": "2026-09-30T17:18:16.6672625Z",
      "endTime": "2026-09-30T17:42:26.002126676Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1cjpvuq75dcn2gn08v3gqss4ksup still exists (deletion dispatched at 2026-09-30T17:18:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010c0210>: 
    resource group "private-keyvault-9g9ksdvdr2w9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-9g9ksdvdr2w9" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2",
      "name": "135095ef-4130-48de-a890-36a3f2ec4ba2",
      "status": "Failed",
      "startTime": "2026-09-30T17:18:16.6672625Z",
      "endTime": "2026-09-30T17:42:26.002126676Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1cjpvuq75dcn2gn08v3gqss4ksup still exists (deletion dispatched at 2026-09-30T17:18:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-m98nfkv66wgc/cluster-candidate-4-23-2zm5lz should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013ea120>: 
    failed to create HCP cluster cluster-candidate-4-23-2zm5lz: failed waiting for cluster="cluster-candidate-4-23-2zm5lz" in resourcegroup="rg-candidate-4-23-m98nfkv66wgc" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59",
      "name": "7e5bac24-0540-4cfd-8079-7530283faa59",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.678276861Z",
      "endTime": "2026-09-30T17:21:45.946479348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-m98nfkv66wgc/cluster-candidate-4-23-2zm5lz should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013ea120>: 
    failed to create HCP cluster cluster-candidate-4-23-2zm5lz: failed waiting for cluster="cluster-candidate-4-23-2zm5lz" in resourcegroup="rg-candidate-4-23-m98nfkv66wgc" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59",
      "name": "7e5bac24-0540-4cfd-8079-7530283faa59",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.678276861Z",
      "endTime": "2026-09-30T17:21:45.946479348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0029c4180>: 
    resource group "feature-aggregation-bffzrcs9xcvk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-bffzrcs9xcvk" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "name": "b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "status": "Failed",
      "startTime": "2026-09-30T17:34:12.215869805Z",
      "endTime": "2026-09-30T17:58:12.821045662Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1rm3k8s9324kfkruicq09u1qjvn0 still exists (deletion dispatched at 2026-09-30T17:34:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0029c4180>: 
    resource group "feature-aggregation-bffzrcs9xcvk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-bffzrcs9xcvk" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "name": "b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "status": "Failed",
      "startTime": "2026-09-30T17:34:12.215869805Z",
      "endTime": "2026-09-30T17:58:12.821045662Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1rm3k8s9324kfkruicq09u1qjvn0 still exists (deletion dispatched at 2026-09-30T17:34:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f1a380>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rwh7s9rvp89r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "name": "eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:15.764760069Z",
      "endTime": "2026-09-30T17:21:16.086415783Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 24m59s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f1a380>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rwh7s9rvp89r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "name": "eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:15.764760069Z",
      "endTime": "2026-09-30T17:21:16.086415783Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 24m59s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0001aacc0>: 
    resource group "rg-hs-pre--r48mqt9lrw4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-fqsxq9" in resourcegroup="rg-hs-pre--r48mqt9lrw4s" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49c44789-52a8-49c2-b053-4835c3571170
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49c44789-52a8-49c2-b053-4835c3571170",
      "name": "49c44789-52a8-49c2-b053-4835c3571170",
      "status": "Failed",
      "startTime": "2026-09-30T17:16:19.615591526Z",
      "endTime": "2026-09-30T17:40:25.981909891Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0hjlm2j48ooq92uj0su5rbnsq03t still exists (deletion dispatched at 2026-09-30T17:16:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0001aacc0>: 
    resource group "rg-hs-pre--r48mqt9lrw4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-fqsxq9" in resourcegroup="rg-hs-pre--r48mqt9lrw4s" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49c44789-52a8-49c2-b053-4835c3571170
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49c44789-52a8-49c2-b053-4835c3571170",
      "name": "49c44789-52a8-49c2-b053-4835c3571170",
      "status": "Failed",
      "startTime": "2026-09-30T17:16:19.615591526Z",
      "endTime": "2026-09-30T17:40:25.981909891Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0hjlm2j48ooq92uj0su5rbnsq03t still exists (deletion dispatched at 2026-09-30T17:16:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0010a8280>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ztrwgb4tgx4r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "name": "fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.354643704Z",
      "endTime": "2026-09-30T17:21:45.995494198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m45s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0010a8280>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ztrwgb4tgx4r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "name": "fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.354643704Z",
      "endTime": "2026-09-30T17:21:45.995494198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m45s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:15:29Z
  Ended:       2026-09-30T17:26:26Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="prometheus", node="aks-userswft1-23433410-vmss000001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-23433410-vmss000001 is unreachable and some workloads may be rescheduled.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:16:26Z
  Ended:       2026-09-30T17:26:26Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="prometheus", node="aks-userswft1-23433410-vmss000001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-23433410-vmss000001 is unreachable and some workloads may be rescheduled.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:40:28Z
  Ended:       2026-09-30T18:18:32Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0hjlm2j48ooq92uj0su5rbnsq03t-x3y7i6e5z1z9c0c", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t5f0hjlm2j48ooq92uj0su5rbnsq03t-x3y7i6e5z1z9c0c (management cluster ci01-j5446016-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:27:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="velero", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:27:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:26:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:26:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="velero", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="set-kubelet-parameters-for-scale", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/set-kubelet-parameters-for-scale has not finished or progressed for at least 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:22:35Z
  Ended:       2026-09-30T17:57:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446016-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-696bd4cfc8-xrrs2", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-696bd4cfc8-xrrs2 (namespace aro-hcp) on cluster ci01-j5446016-svc has averaged 170% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:22:35Z
  Ended:       2026-09-30T18:09:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446016-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-696bd4cfc8-bzvvh", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-696bd4cfc8-bzvvh (namespace aro-hcp) on cluster ci01-j5446016-svc has averaged 209.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 93 time(s) (25 unknown, 68 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:04:15Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="etcd-2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0e8dbaqbo8t5dt5vdcasi8k08a4q-v2v3z7b8u7q0n1w", pod="kube-state-metrics-hcp-754f855b69-w9pxh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0e8dbaqbo8t5dt5vdcasi8k08a4q-v2v3z7b8u7q0n1w/kube-state-metrics-hcp-754f855b69-w9pxh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:34:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="router-79bddd4695-4xzx4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/router-79bddd4695-4xzx4 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="multus-admission-controller-77fb5c8d8f-hmg8g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/multus-admission-controller-77fb5c8d8f-hmg8g has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:34:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="router-6c4c4f9b46-9jc7h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/router-6c4c4f9b46-9jc7h has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="network-node-identity-7b6f464c54-ttl5f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/network-node-identity-7b6f464c54-ttl5f has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lqugf7p62g2vu15647hciuo7rqe-ephemeral-disk", pod="kube-state-metrics-hcp-754f855b69-5lx7w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lqugf7p62g2vu15647hciuo7rqe-ephemeral-disk/kube-state-metrics-hcp-754f855b69-5lx7w has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f1mqu517if66tcq7vv1883rm6svr2-x0m6x6k6h4c6v2j", pod="kube-state-metrics-hcp-754f855b69-cvfnm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f1mqu517if66tcq7vv1883rm6svr2-x0m6x6k6h4c6v2j/kube-state-metrics-hcp-754f855b69-cvfnm has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f19n96q5slf0m0lsru0ta7ea87dun-h5n4x4c7s1h6n9x", pod="kube-state-metrics-hcp-754f855b69-5ds5m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f19n96q5slf0m0lsru0ta7ea87dun-h5n4x4c7s1h6n9x/kube-state-metrics-hcp-754f855b69-5ds5m has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="ovnkube-control-plane-549bdb54d-n62hb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/ovnkube-control-plane-549bdb54d-n62hb has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-30T17:06:13Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress", pod="network-node-identity-5469fd6495-rbqgx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress/network-node-identity-5469fd6495-rbqgx has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-30T17:06:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="kube-controller-manager-59b5946b4c-8dnp4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/kube-controller-manager-59b5946b4c-8dnp4 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-30T17:08:15Z
  Ended:       2026-09-30T17:14:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0nnvckp9ok2ivvajoua5mi6rfcis-e4a2q0w1u4z3v6w", pod="openshift-apiserver-5b6785f9dc-fc7tr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0nnvckp9ok2ivvajoua5mi6rfcis-e4a2q0w1u4z3v6w/openshift-apiserver-5b6785f9dc-fc7tr has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-30T17:08:15Z
  Ended:       2026-09-30T17:14:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f15npkrjd452rtt7ar07bisihfskg-j2c7h4h2c6p4s3p", pod="openshift-apiserver-7dd55877cd-gkcd5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f15npkrjd452rtt7ar07bisihfskg-j2c7h4h2c6p4s3p/openshift-apiserver-7dd55877cd-gkcd5 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-30T17:08:15Z
  Ended:       2026-09-30T17:14:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0uvlpj2ovpuf5b8grf3i9edvpodq-g3v6g1h5e0h5l2j", pod="openshift-apiserver-758c4878d9-7j48d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0uvlpj2ovpuf5b8grf3i9edvpodq-g3v6g1h5e0h5l2j/openshift-apiserver-758c4878d9-7j48d has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-30T17:11:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress", pod="network-node-identity-5469fd6495-f7zgd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress/network-node-identity-5469fd6495-f7zgd has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-30T17:11:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="router-8565cb6474-894zb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/router-8565cb6474-894zb has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-30T17:11:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="kube-apiserver-674f58879-j82fb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/kube-apiserver-674f58879-j82fb has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="network-node-identity-7b6f464c54-gjfm6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/network-node-identity-7b6f464c54-gjfm6 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="openshift-apiserver-f89c544f9-jcfcs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/openshift-apiserver-f89c544f9-jcfcs has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v", pod="openshift-apiserver-5d4fc6f8c7-68m94", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v/openshift-apiserver-5d4fc6f8c7-68m94 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v", pod="kube-apiserver-87c8b5fd9-ph74m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v/kube-apiserver-87c8b5fd9-ph74m has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-30T17:13:16Z
  Ended:       2026-09-30T17:22:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-p75kr", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-p75kr has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:13:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-xfhsn", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-xfhsn has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:13:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="set-kubelet-parameters-for-scale-vdkwr", region="westus3", severity="warning"
  Description: Pod kube-system/set-kubelet-parameters-for-scale-vdkwr has been in a non-ready state for longer than 5 minutes.

alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000ca3800>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-xw6tpr5d2lsl" to finish creating: GET https://rp.j4862592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "name": "3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "status": "Failed",
      "startTime": "2026-09-30T17:03:28.253892004Z",
      "endTime": "2026-09-30T17:22:36.043787859Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000ca3800>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-xw6tpr5d2lsl" to finish creating: GET https://rp.j4862592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "name": "3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "status": "Failed",
      "startTime": "2026-09-30T17:03:28.253892004Z",
      "endTime": "2026-09-30T17:22:36.043787859Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-xnh68bcrp77f/cluster-candidate-4-21-944b8p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000874f60>: 
    failed to create HCP cluster cluster-candidate-4-21-944b8p: failed waiting for cluster="cluster-candidate-4-21-944b8p" in resourcegroup="rg-candidate-4-21-xnh68bcrp77f" to finish creating: GET https://rp.j5828352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0796d72b-193f-46f8-9e78-aaf27448038b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0796d72b-193f-46f8-9e78-aaf27448038b",
      "name": "0796d72b-193f-46f8-9e78-aaf27448038b",
      "status": "Failed",
      "startTime": "2026-09-30T15:56:05.509326654Z",
      "endTime": "2026-09-30T16:25:07.850265278Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-xnh68bcrp77f/cluster-candidate-4-21-944b8p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000874f60>: 
    failed to create HCP cluster cluster-candidate-4-21-944b8p: failed waiting for cluster="cluster-candidate-4-21-944b8p" in resourcegroup="rg-candidate-4-21-xnh68bcrp77f" to finish creating: GET https://rp.j5828352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0796d72b-193f-46f8-9e78-aaf27448038b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0796d72b-193f-46f8-9e78-aaf27448038b",
      "name": "0796d72b-193f-46f8-9e78-aaf27448038b",
      "status": "Failed",
      "startTime": "2026-09-30T15:56:05.509326654Z",
      "endTime": "2026-09-30T16:25:07.850265278Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T16:05:30Z
  Ended:       2026-09-30T16:33:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5828352-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5e65ks30us7u7a3qg1vdbknb1vvqs6-t3z0s0t0v4e0p5i", pod="router-865f6cd877-rnq4k", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5e65ks30us7u7a3qg1vdbknb1vvqs6-t3z0s0t0v4e0p5i/router-865f6cd877-rnq4k has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T16:07:32Z
  Ended:       2026-09-30T16:25:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5828352-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5e65gj37b763s4sdv3kievh5fge8ub-c7d3y8f1x8e6t7o", pod="router-8f95b5bf6-pw2hq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5e65gj37b763s4sdv3kievh5fge8ub-c7d3y8f1x8e6t7o/router-8f95b5bf6-pw2hq has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T15:25:46Z
  Ended:       2026-09-30T15:31:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8561152-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0001f61c0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-2ggb82wh5r22" to finish creating: GET https://rp.j4139264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50346f69-92e6-45cd-a218-e9b1ff541e03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50346f69-92e6-45cd-a218-e9b1ff541e03",
      "name": "50346f69-92e6-45cd-a218-e9b1ff541e03",
      "status": "Failed",
      "startTime": "2026-09-30T15:09:42.512906803Z",
      "endTime": "2026-09-30T15:38:51.371261651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0001f61c0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-2ggb82wh5r22" to finish creating: GET https://rp.j4139264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50346f69-92e6-45cd-a218-e9b1ff541e03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50346f69-92e6-45cd-a218-e9b1ff541e03",
      "name": "50346f69-92e6-45cd-a218-e9b1ff541e03",
      "status": "Failed",
      "startTime": "2026-09-30T15:09:42.512906803Z",
      "endTime": "2026-09-30T15:38:51.371261651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T15:21:57Z
  Ended:       2026-09-30T15:50:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4139264-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5dgduvrdd5glf5k0m6vesv46nt0jhm-private-kas", pod="router-55bfdfc49b-qhf45", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5dgduvrdd5glf5k0m6vesv46nt0jhm-private-kas/router-55bfdfc49b-qhf45 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c9ee28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c9ee28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T15:24:36Z
  Ended:       2026-09-30T15:30:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9645824-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5d7bu25qfk2cfhnp3n9qlq37dgcjbd-private-ingress", pod="etcd-0", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5d7bu25qfk2cfhnp3n9qlq37dgcjbd-private-ingress/etcd-0 has been in a non-ready state for longer than 5 minutes.

time=2026-09-30T14:35:13.661Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T14:35:14.082Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j1358720-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j1358720-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e962c7b1-5eb5-497b-9341-4b3a121f354e region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e962c7b1-5eb5-497b-9341-4b3a121f354e INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T14:35:14.083Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:35:14.275Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T14:35:14.491Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T14:35:14.491Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:35:14.524Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:45:17.368Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:45:17.369Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:46:17.370Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T14:46:17.815Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j1358720-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j1358720-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e962c7b1-5eb5-497b-9341-4b3a121f354e region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e962c7b1-5eb5-497b-9341-4b3a121f354e INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T14:46:17.815Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:18.042Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T14:46:18.350Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:18.350Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:18.601Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T14:46:19.301Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:19.301Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:19.487Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:19.912Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T14:46:20.506Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T14:46:20.869Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:21.189Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:21.550Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:21.905Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T14:46:21.905Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:21.951Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T14:56:23.257Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:56:23.257Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:57:23.259Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T14:57:24.108Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j1358720-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j1358720-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e962c7b1-5eb5-497b-9341-4b3a121f354e region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e962c7b1-5eb5-497b-9341-4b3a121f354e INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T14:57:24.108Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:24.411Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T14:57:24.749Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:24.749Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:25.002Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T14:57:25.620Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:25.620Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:25.977Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:26.226Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T14:57:26.765Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T14:57:26.974Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:27.201Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:27.763Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:28.017Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T14:57:28.017Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:28.079Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T15:05:13.695Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T15:05:13.695Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:52:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1358720-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.42:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1358720/providers/microsoft.monitor/accounts/services-j1358720", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-r6958", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:52:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1358720-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.42:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1358720/providers/microsoft.monitor/accounts/services-j1358720", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-r6958", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:42:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1358720-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-png7s", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-png7s has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:43:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1358720-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-bzbm7", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-bzbm7 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:46:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1358720-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-mkkjw", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-mkkjw has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:51:34Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j1358720-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j1358720-svc has been unreachable for 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:242]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc000b4a880>: 
    failed starting cluster creation "acr-pull-create" in resourcegroup="acr-pull-create-v5r8zv9r2jvq": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/acr-pull-create
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi\": containerRegistry requires cluster version 4.22.0 or above",
        "target": "properties.platform.containerRegistry.managedIdentity"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:242]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc000b4a880>: 
    failed starting cluster creation "acr-pull-create" in resourcegroup="acr-pull-create-v5r8zv9r2jvq": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/acr-pull-create
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi\": containerRegistry requires cluster version 4.22.0 or above",
        "target": "properties.platform.containerRegistry.managedIdentity"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:96]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c48700>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-7mqkjzxbfvhm": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-7mqkjzxbfvhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"Private\": not supported when customerProperties.version.id is below 4.22",
        "target": "properties.api.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:96]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c48700>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-7mqkjzxbfvhm": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-7mqkjzxbfvhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"Private\": not supported when customerProperties.version.id is below 4.22",
        "target": "properties.api.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:191]: failed to update cluster with new KMS key
Unexpected error:
    <*exported.ResponseError | 0xc000ece360>: 
    PATCH https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kms-key-rotate-s7rqrhtvbcff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/kms-key-rotate-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"484184ba23584b37813d670c444e552b\": KMS key version rotation requires cluster version 4.22.0 or above",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.activeKey.version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:191]: failed to update cluster with new KMS key
Unexpected error:
    <*exported.ResponseError | 0xc000ece360>: 
    PATCH https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kms-key-rotate-s7rqrhtvbcff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/kms-key-rotate-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"484184ba23584b37813d670c444e552b\": KMS key version rotation requires cluster version 4.22.0 or above",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.activeKey.version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000ea00e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-7mqggzhqq55s" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90",
      "name": "1ec169ee-a122-45a3-a622-b7083455fb90",
      "status": "Failed",
      "startTime": "2026-09-30T14:16:43.796681639Z",
      "endTime": "2026-09-30T14:35:52.497968724Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000ea00e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-7mqggzhqq55s" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90",
      "name": "1ec169ee-a122-45a3-a622-b7083455fb90",
      "status": "Failed",
      "startTime": "2026-09-30T14:16:43.796681639Z",
      "endTime": "2026-09-30T14:35:52.497968724Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000686fa0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-hqw2kg" in resourcegroup="rg-np-downgrade-hqw2kg-tzshqkp8zjqk" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "name": "0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "status": "Failed",
      "startTime": "2026-09-30T14:18:31.838214826Z",
      "endTime": "2026-09-30T14:37:32.428641421Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000686fa0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-hqw2kg" in resourcegroup="rg-np-downgrade-hqw2kg-tzshqkp8zjqk" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "name": "0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "status": "Failed",
      "startTime": "2026-09-30T14:18:31.838214826Z",
      "endTime": "2026-09-30T14:37:32.428641421Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-889bmn" for cluster "cluster-candidate-4-20-889bmn"
Unexpected error:
    <*fmt.wrapError | 0xc0015841e0>: 
    failed to create NodePool np-889bmn: failed waiting for nodepool="np-889bmn" for cluster "cluster-candidate-4-20-889bmn" in resourcegroup="rg-candidate-4-20-qcvff5h4wj94" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "name": "e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "status": "Failed",
      "startTime": "2026-09-30T12:19:11.883646471Z",
      "endTime": "2026-09-30T12:38:11.728721311Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-889bmn" for cluster "cluster-candidate-4-20-889bmn"
Unexpected error:
    <*fmt.wrapError | 0xc0015841e0>: 
    failed to create NodePool np-889bmn: failed waiting for nodepool="np-889bmn" for cluster "cluster-candidate-4-20-889bmn" in resourcegroup="rg-candidate-4-20-qcvff5h4wj94" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "name": "e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "status": "Failed",
      "startTime": "2026-09-30T12:19:11.883646471Z",
      "endTime": "2026-09-30T12:38:11.728721311Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:615]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:615]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000c681a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-fl5kpw" in resourcegroup="rg-np-version-upgrade-fl5kpw-96mvvhr695cd" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "name": "a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "status": "Failed",
      "startTime": "2026-09-30T12:36:12.039132485Z",
      "endTime": "2026-09-30T12:55:21.071689427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000c681a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-fl5kpw" in resourcegroup="rg-np-version-upgrade-fl5kpw-96mvvhr695cd" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "name": "a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "status": "Failed",
      "startTime": "2026-09-30T12:36:12.039132485Z",
      "endTime": "2026-09-30T12:55:21.071689427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:158]: expected all nodes to have label 'key1=value1'
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:158]: expected all nodes to have label 'key1=value1'
Expected
    <bool>: ...
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000abb5a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dr2mph" in resourcegroup="rg-np-noedge-dr2mph-j7ng55t8mcf6" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611",
      "name": "78b330e2-3532-4f23-91f1-b24676955611",
      "status": "Failed",
      "startTime": "2026-09-30T12:38:52.851263491Z",
      "endTime": "2026-09-30T12:58:01.083392787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000abb5a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dr2mph" in resourcegroup="rg-np-noedge-dr2mph-j7ng55t8mcf6" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611",
      "name": "78b330e2-3532-4f23-91f1-b24676955611",
      "status": "Failed",
      "startTime": "2026-09-30T12:38:52.851263491Z",
      "endTime": "2026-09-30T12:58:01.083392787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:476]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:476]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00082e468>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00082e468>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:755]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:755]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m"
Unexpected error:
    <*fmt.wrapError | 0xc000f7d6c0>: 
    failed to create NodePool np-wsv85m: failed waiting for nodepool="np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m" in resourcegroup="rg-candidate-4-23-x5s4hpmkwzhp" to finish creating: GET https://rp.j1071872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550",
      "name": "edf97b61-1366-48f5-936f-5378dbccb550",
      "status": "Failed",
      "startTime": "2026-09-30T11:35:07.81865953Z",
      "endTime": "2026-09-30T11:54:13.175354213Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m"
Unexpected error:
    <*fmt.wrapError | 0xc000f7d6c0>: 
    failed to create NodePool np-wsv85m: failed waiting for nodepool="np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m" in resourcegroup="rg-candidate-4-23-x5s4hpmkwzhp" to finish creating: GET https://rp.j1071872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550",
      "name": "edf97b61-1366-48f5-936f-5378dbccb550",
      "status": "Failed",
      "startTime": "2026-09-30T11:35:07.81865953Z",
      "endTime": "2026-09-30T11:54:13.175354213Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T11:34:50Z
  Ended:       2026-09-30T11:49:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1071872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5a636m6sfjl130rh16efne40festfg-cilium-cluster", pod="router-7f64d88865-r8tm4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5a636m6sfjl130rh16efne40festfg-cilium-cluster/router-7f64d88865-r8tm4 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T11:35:51Z
  Ended:       2026-09-30T11:49:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1071872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5a636to94rd3t38eir1d61k2r3hvq2-m4p7l3e8z9w1p5p", pod="router-5cc8888886-kjp9p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5a636to94rd3t38eir1d61k2r3hvq2-m4p7l3e8z9w1p5p/router-5cc8888886-kjp9p has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T12:05:05Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9346176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.63:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9346176/providers/microsoft.monitor/accounts/services-j9346176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c94b5c67c-s5vnr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9346176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T12:05:05Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9346176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.63:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9346176/providers/microsoft.monitor/accounts/services-j9346176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c94b5c67c-s5vnr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9346176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000bd4f00>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-l57gw6mdwbj6" to finish creating: GET https://rp.j2690432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75db90be-5602-433c-8319-190a1fa900d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75db90be-5602-433c-8319-190a1fa900d9",
      "name": "75db90be-5602-433c-8319-190a1fa900d9",
      "status": "Failed",
      "startTime": "2026-09-30T10:22:37.553403502Z",
      "endTime": "2026-09-30T10:41:41.061493076Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-l57gw6mdwbj6--managed/providers/Microsoft.Compute/virtualMachines/d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-l57gw6mdwbj6--managed/providers/Microsoft.Compute/virtualMachines/d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000bd4f00>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-l57gw6mdwbj6" to finish creating: GET https://rp.j2690432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75db90be-5602-433c-8319-190a1fa900d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75db90be-5602-433c-8319-190a1fa900d9",
      "name": "75db90be-5602-433c-8319-190a1fa900d9",
      "status": "Failed",
      "startTime": "2026-09-30T10:22:37.553403502Z",
      "endTime": "2026-09-30T10:41:41.061493076Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-l57gw6mdwbj6--managed/providers/Microsoft.Compute/virtualMachines/d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-l57gw6mdwbj6--managed/providers/Microsoft.Compute/virtualMachines/d1u4m0g7s7a4h7b-nodepool-128gib-pj9gv-krcrc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-30T10:02:16.720Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T10:02:17.023Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j5090432-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j5090432-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e5a2e4f0-d338-424f-81b6-3ea36e90ffa8 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e5a2e4f0-d338-424f-81b6-3ea36e90ffa8 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T10:02:17.023Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:02:17.180Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T10:02:17.354Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T10:02:17.354Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:02:17.402Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:12:19.938Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T10:12:19.938Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T10:13:19.944Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T10:13:20.698Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j5090432-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j5090432-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e5a2e4f0-d338-424f-81b6-3ea36e90ffa8 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e5a2e4f0-d338-424f-81b6-3ea36e90ffa8 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T10:13:20.699Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:13:21.000Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T10:13:21.389Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:13:21.389Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:13:21.559Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T10:13:22.194Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:13:22.194Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:13:22.766Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:13:23.138Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T10:13:23.614Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T10:13:24.021Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:13:24.220Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:13:24.545Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:13:24.923Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T10:13:24.923Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:13:25.004Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T10:23:26.540Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T10:23:26.540Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T10:24:26.541Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T10:24:26.978Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j5090432-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j5090432-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e5a2e4f0-d338-424f-81b6-3ea36e90ffa8 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e5a2e4f0-d338-424f-81b6-3ea36e90ffa8 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T10:24:26.978Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:24:27.205Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T10:24:27.597Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:24:27.597Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:24:27.807Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T10:24:28.587Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:24:28.587Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:24:28.952Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:24:29.210Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T10:24:29.577Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T10:24:29.759Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:24:30.112Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:24:30.448Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T10:24:30.762Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T10:24:30.762Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T10:24:30.844Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T10:32:16.654Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T10:32:16.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:19:02Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j5090432-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j5090432-svc has been unreachable for 15 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:08:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5090432-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-c997l", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-c997l has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:08:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5090432-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-qptj8", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-qptj8 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:08:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5090432-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-69sbs", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-69sbs has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:18:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5090432-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.69:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5090432/providers/microsoft.monitor/accounts/services-j5090432", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-gmtlp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:18:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5090432-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.69:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5090432/providers/microsoft.monitor/accounts/services-j5090432", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-gmtlp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 611.396s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00158e610>: 
    run command "e2e-runcommand-1790764865640292876" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 611.396s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00158e610>: 
    run command "e2e-runcommand-1790764865640292876" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred

time=2026-09-30T10:09:47.479Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-09-30T10:09:48.123Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4]"
time=2026-09-30T10:09:48.123Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-30T10:09:48.426Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-09-30T10:09:48.816Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-09-30T10:09:48.816Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-30T10:09:48.942Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-30T10:14:53.371Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-09-30T10:14:53.371Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-7jbkqj" with version 4.21.34 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011488e0>: 
    failed to create HCP cluster cluster-zstream-4-21-7jbkqj: failed waiting for cluster="cluster-zstream-4-21-7jbkqj" in resourcegroup="rg-zstream-upgrade-4-21-7qqcshqrm588" to finish creating: GET https://rp.j7585920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-7jbkqj" with version 4.21.34 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011488e0>: 
    failed to create HCP cluster cluster-zstream-4-21-7jbkqj: failed waiting for cluster="cluster-zstream-4-21-7jbkqj" in resourcegroup="rg-zstream-upgrade-4-21-7qqcshqrm588" to finish creating: GET https://rp.j7585920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xkg7gf" for cluster "cluster-candidate-4-20-xkg7gf"
Unexpected error:
    <*fmt.wrapError | 0xc001435b60>: 
    failed to create NodePool np-xkg7gf: failed waiting for nodepool="np-xkg7gf" for cluster "cluster-candidate-4-20-xkg7gf" in resourcegroup="rg-candidate-4-20-j47p9rbxx46g" to finish creating: GET https://rp.j5597824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26468792-a524-4bf9-970b-d4edc2f9bfe8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26468792-a524-4bf9-970b-d4edc2f9bfe8",
      "name": "26468792-a524-4bf9-970b-d4edc2f9bfe8",
      "status": "Failed",
      "startTime": "2026-09-30T10:25:29.553885983Z",
      "endTime": "2026-09-30T10:44:30.082076643Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-j47p9rbxx46g-candidate-xkg7gf--managed/providers/Microsoft.Compute/virtualMachines/q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-j47p9rbxx46g-candidate-xkg7gf--managed/providers/Microsoft.Compute/virtualMachines/q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xkg7gf" for cluster "cluster-candidate-4-20-xkg7gf"
Unexpected error:
    <*fmt.wrapError | 0xc001435b60>: 
    failed to create NodePool np-xkg7gf: failed waiting for nodepool="np-xkg7gf" for cluster "cluster-candidate-4-20-xkg7gf" in resourcegroup="rg-candidate-4-20-j47p9rbxx46g" to finish creating: GET https://rp.j5597824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26468792-a524-4bf9-970b-d4edc2f9bfe8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26468792-a524-4bf9-970b-d4edc2f9bfe8",
      "name": "26468792-a524-4bf9-970b-d4edc2f9bfe8",
      "status": "Failed",
      "startTime": "2026-09-30T10:25:29.553885983Z",
      "endTime": "2026-09-30T10:44:30.082076643Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-j47p9rbxx46g-candidate-xkg7gf--managed/providers/Microsoft.Compute/virtualMachines/q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-j47p9rbxx46g-candidate-xkg7gf--managed/providers/Microsoft.Compute/virtualMachines/q2o4d5p5y8u3o9e-np-xkg7gf-jxzwr-xwjrg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-30T10:01:25.496Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-09-30T10:01:25.890Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperPoolCert-34 clientId:72671881-beb9-4028-bc7c-47f511281566 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpci01oidcj2121984.z9.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj2121984.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] backupCadenceProfile:testing backupScheduleState:Disabled clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:072ccd06-457a-49cc-a612-ad4dbbc199a3 databaseName:arohcpci01-rp-j2121984 databaseUrl:https://arohcpci01-rp-j2121984.documents.azure.com:443/ kubeApplierCosmosContainerName:Manifests-MC-1 location:canadacentral] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-gq2sycdx/pipeline@sha256:ec35d9956c06798bf0f6c7e4116d8b1c120ff9a5a7cbe4e991d36cae1bea464e replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:700m memory:1Gi]] zoneCount:3] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] logVerbosity:4 maestro:map[sourceEnvironmentIdentifier:arohcpci01] miMock:map[certName:msiMockPoolCert-18 clientId:8c088886-a118-4973-8168-0d7ae1493aa7 clientIdSecretName: principalId:ccbf438f-08ec-4e2b-ae4e-219ce7dc3762 principalIdSecretName: tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-gq2sycdx/pipeline:pull workloadIdentityClientId:7cbf62ec-0a3a-4039-a0c0-7c88aa6f75bf workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:072ccd06-457a-49cc-a612-ad4dbbc199a3 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv]]"
time=2026-09-30T10:01:25.890Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-30T10:01:26.074Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-09-30T10:01:26.234Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-09-30T10:01:26.234Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-30T10:01:26.298Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-30T10:06:31.687Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 0/2\ncontext deadline exceeded"
time=2026-09-30T10:06:31.687Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 0/2\ncontext deadline exceeded"

alert fired 95 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-rsvtx", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-rsvtx has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-796x2", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-796x2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-pcjcl", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-pcjcl has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-z998l", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-z998l has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-m8698", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-m8698 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-82cr4", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-82cr4 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-pzjb4", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-pzjb4 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-7x96q", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-7x96q has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-x2wph", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-x2wph has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-n9dj7", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-n9dj7 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-nwhcl", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-nwhcl has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-vgjxq", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-vgjxq has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-5nrdf", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-5nrdf has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-m52wd", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-m52wd has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-5g7dm", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-5g7dm has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-sqlsb", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-sqlsb has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-vfwjz", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-vfwjz has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-czw9d", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-czw9d has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-ft9sh", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-ft9sh has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-2lclr", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-2lclr has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-tgq28", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-tgq28 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-qvg8l", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-qvg8l has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-js2ms", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-js2ms has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-s2mtg", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-s2mtg has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-g76kv", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-g76kv has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-6rcxw", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-6rcxw has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-sw7b9", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-sw7b9 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-c4n6b", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-c4n6b has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-lq6ck", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-lq6ck has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-dvlrh", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-dvlrh has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-cj9md", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-cj9md has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-bjcl4", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-bjcl4 has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-48gt2", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-48gt2 has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-qchw5", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-qchw5 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-bz68s", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-bz68s has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-tv8hg", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-tv8hg has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-8n25z", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-8n25z has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-n4r6z", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-n4r6z has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-gpj9m", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-gpj9m has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-f4z6c", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-f4z6c has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-cd9g2", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-cd9g2 has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-vqj8z", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-vqj8z has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-zm597", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-zm597 has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-q6m5f", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-q6m5f has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-bxhlp", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-bxhlp has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-8c6pj", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-8c6pj has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-gjs74", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-gjs74 has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-lrcrl", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-lrcrl has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-kwfr5", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-kwfr5 has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-jnhrj", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-jnhrj has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-j2x54", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-j2x54 has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-m4w5z", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-m4w5z has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-c5jzp", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-c5jzp has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-xtl8c", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-xtl8c has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-pk965", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-pk965 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-xc2p8", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-xc2p8 has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-jkhgh", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-jkhgh has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-vl6w8", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-vl6w8 has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-9xhbm", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-9xhbm has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-pldcj", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-pldcj has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-wzvnf", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-wzvnf has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-c8pn2", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-c8pn2 has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-7dgv9", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-7dgv9 has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-cw6k5", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-cw6k5 has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-cp898", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-cp898 has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-vfv5j", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-vfv5j has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-mcbpb", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-mcbpb has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-fq9qk", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-fq9qk has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-v2www", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-v2www has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-8j9r8", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-8j9r8 has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-dtv87", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-dtv87 has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-pk6pc", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-pk6pc has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-jpfz4", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-jpfz4 has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-v6ml2", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-v6ml2 has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-msxpq", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-msxpq has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-hbpsf", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-hbpsf has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-5njm9", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-5njm9 has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-2nc5h", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-2nc5h has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-7ftmt", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-7ftmt has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-ccxrv", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-ccxrv has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-ngg9z", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-ngg9z has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-8w6bk", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-8w6bk has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-jgzjj", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-jgzjj has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-4dchf", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-4dchf has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-dkq6z", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-dkq6z has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-vvrhd", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-vvrhd has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-fllz9", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-fllz9 has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-jckvz", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-jckvz has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-2fn44", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-2fn44 has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-k5psj", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-k5psj has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-4r64q", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-4r64q has been in a non-ready state for longer than 5 minutes.
Firing 92:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-98h9z", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-98h9z has been in a non-ready state for longer than 5 minutes.
Firing 93:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-9b6sv", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-9b6sv has been in a non-ready state for longer than 5 minutes.
Firing 94:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-sfw6f", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-sfw6f has been in a non-ready state for longer than 5 minutes.
Firing 95:
  State:       Fired (not resolved)
  Started:     2026-09-30T10:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2121984-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-6285h", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-6285h has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002511248>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xk42m82bkxtn" to finish creating: GET https://rp.j2567168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/034a0762-ab5c-48da-87dd-8d208861a970
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/034a0762-ab5c-48da-87dd-8d208861a970",
      "name": "034a0762-ab5c-48da-87dd-8d208861a970",
      "status": "Failed",
      "startTime": "2026-09-30T08:55:45.833258741Z",
      "endTime": "2026-09-30T09:14:52.939311971Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-nssc6-7nz65: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xk42m82bkxtn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-nssc6-7nz65 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-nssc6-7nz65: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xk42m82bkxtn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-nssc6-7nz65 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-nssc6-7nz65": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-nssc6-7nz65" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xk42m82bkxtn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-nssc6-7nz65/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-nssc6-7nz65' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002511248>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xk42m82bkxtn" to finish creating: GET https://rp.j2567168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/034a0762-ab5c-48da-87dd-8d208861a970
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/034a0762-ab5c-48da-87dd-8d208861a970",
      "name": "034a0762-ab5c-48da-87dd-8d208861a970",
      "status": "Failed",
      "startTime": "2026-09-30T08:55:45.833258741Z",
      "endTime": "2026-09-30T09:14:52.939311971Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-nssc6-7nz65: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xk42m82bkxtn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-nssc6-7nz65 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-nssc6-7nz65: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xk42m82bkxtn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-nssc6-7nz65 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-nssc6-7nz65": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-nssc6-7nz65" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xk42m82bkxtn--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-nssc6-7nz65/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-nssc6-7nz65' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000aae2c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-nkgjwtvv229n" to finish creating: GET https://rp.j8181888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc933c7-0d41-4651-a4a8-b2c8e1264ee6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc933c7-0d41-4651-a4a8-b2c8e1264ee6",
      "name": "4fc933c7-0d41-4651-a4a8-b2c8e1264ee6",
      "status": "Failed",
      "startTime": "2026-09-30T08:33:01.043737315Z",
      "endTime": "2026-09-30T08:52:07.873742657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-nkgjwtvv229n--managed/providers/Microsoft.Compute/virtualMachines/j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-nkgjwtvv229n--managed/providers/Microsoft.Compute/virtualMachines/j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000aae2c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-nkgjwtvv229n" to finish creating: GET https://rp.j8181888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc933c7-0d41-4651-a4a8-b2c8e1264ee6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc933c7-0d41-4651-a4a8-b2c8e1264ee6",
      "name": "4fc933c7-0d41-4651-a4a8-b2c8e1264ee6",
      "status": "Failed",
      "startTime": "2026-09-30T08:33:01.043737315Z",
      "endTime": "2026-09-30T08:52:07.873742657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-nkgjwtvv229n--managed/providers/Microsoft.Compute/virtualMachines/j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-nkgjwtvv229n--managed/providers/Microsoft.Compute/virtualMachines/j6f0a3z3m5a4v2o-np-1-cdxdq-nmfh9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c97540>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4vhtkz65lcdf" to finish creating: GET https://rp.j6898944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "name": "f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "status": "Failed",
      "startTime": "2026-09-30T07:58:35.462676175Z",
      "endTime": "2026-09-30T08:17:44.959895115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c97540>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4vhtkz65lcdf" to finish creating: GET https://rp.j6898944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "name": "f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "status": "Failed",
      "startTime": "2026-09-30T07:58:35.462676175Z",
      "endTime": "2026-09-30T08:17:44.959895115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014003d8>: 
    resource group "admin-credential-lifecycle-test-5rfj4vwnhrwj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-vxghsm" in resourcegroup="admin-credential-lifecycle-test-5rfj4vwnhrwj" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0efef022-0122-4549-9e52-3393af9918c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0efef022-0122-4549-9e52-3393af9918c1",
      "name": "0efef022-0122-4549-9e52-3393af9918c1",
      "status": "Failed",
      "startTime": "2026-09-30T07:44:51.882854873Z",
      "endTime": "2026-09-30T08:08:52.436086515Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofgpftr9snm78jc1alvisvlihdkq still exists (deletion dispatched at 2026-09-30T07:45:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014003d8>: 
    resource group "admin-credential-lifecycle-test-5rfj4vwnhrwj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-vxghsm" in resourcegroup="admin-credential-lifecycle-test-5rfj4vwnhrwj" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0efef022-0122-4549-9e52-3393af9918c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0efef022-0122-4549-9e52-3393af9918c1",
      "name": "0efef022-0122-4549-9e52-3393af9918c1",
      "status": "Failed",
      "startTime": "2026-09-30T07:44:51.882854873Z",
      "endTime": "2026-09-30T08:08:52.436086515Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofgpftr9snm78jc1alvisvlihdkq still exists (deletion dispatched at 2026-09-30T07:45:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0009f4378>: 
    resource group "v20261001preview-glc4g8rxxg96": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-glc4g8rxxg96" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22941031-f785-42ec-930f-02f9b4baf5df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22941031-f785-42ec-930f-02f9b4baf5df",
      "name": "22941031-f785-42ec-930f-02f9b4baf5df",
      "status": "Failed",
      "startTime": "2026-09-30T07:50:24.203578528Z",
      "endTime": "2026-09-30T08:14:32.552450638Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obpih7ob1nmt9oif2arpkqr6ctja still exists (deletion dispatched at 2026-09-30T07:50:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0009f4378>: 
    resource group "v20261001preview-glc4g8rxxg96": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-glc4g8rxxg96" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22941031-f785-42ec-930f-02f9b4baf5df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22941031-f785-42ec-930f-02f9b4baf5df",
      "name": "22941031-f785-42ec-930f-02f9b4baf5df",
      "status": "Failed",
      "startTime": "2026-09-30T07:50:24.203578528Z",
      "endTime": "2026-09-30T08:14:32.552450638Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obpih7ob1nmt9oif2arpkqr6ctja still exists (deletion dispatched at 2026-09-30T07:50:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001e06228>: 
    resource group "rg-np-version-upgrade-lhpwgz-8wlwbkbk8nl7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-lhpwgz" in resourcegroup="rg-np-version-upgrade-lhpwgz-8wlwbkbk8nl7" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/32343bad-f62c-41f1-a78a-34ece808734d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/32343bad-f62c-41f1-a78a-34ece808734d",
      "name": "32343bad-f62c-41f1-a78a-34ece808734d",
      "status": "Failed",
      "startTime": "2026-09-30T08:09:00.777952873Z",
      "endTime": "2026-09-30T08:33:06.459889594Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oj68ut6sfef5ef3g3kgp5llvpbpr still exists (deletion dispatched at 2026-09-30T08:09:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001e06228>: 
    resource group "rg-np-version-upgrade-lhpwgz-8wlwbkbk8nl7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-lhpwgz" in resourcegroup="rg-np-version-upgrade-lhpwgz-8wlwbkbk8nl7" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/32343bad-f62c-41f1-a78a-34ece808734d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/32343bad-f62c-41f1-a78a-34ece808734d",
      "name": "32343bad-f62c-41f1-a78a-34ece808734d",
      "status": "Failed",
      "startTime": "2026-09-30T08:09:00.777952873Z",
      "endTime": "2026-09-30T08:33:06.459889594Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oj68ut6sfef5ef3g3kgp5llvpbpr still exists (deletion dispatched at 2026-09-30T08:09:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c0e078>: 
    resource group "admin-api-breakglass-bvjprckvbd9j": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-bvjprckvbd9j" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caa65f31-7834-40ae-9558-2b145d3d3707
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caa65f31-7834-40ae-9558-2b145d3d3707",
      "name": "caa65f31-7834-40ae-9558-2b145d3d3707",
      "status": "Failed",
      "startTime": "2026-09-30T07:42:39.486123591Z",
      "endTime": "2026-09-30T08:06:42.572288271Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oguqit52ovjjoaakam9v60fven2c still exists (deletion dispatched at 2026-09-30T07:42:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c0e078>: 
    resource group "admin-api-breakglass-bvjprckvbd9j": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-bvjprckvbd9j" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caa65f31-7834-40ae-9558-2b145d3d3707
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/caa65f31-7834-40ae-9558-2b145d3d3707",
      "name": "caa65f31-7834-40ae-9558-2b145d3d3707",
      "status": "Failed",
      "startTime": "2026-09-30T07:42:39.486123591Z",
      "endTime": "2026-09-30T08:06:42.572288271Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oguqit52ovjjoaakam9v60fven2c still exists (deletion dispatched at 2026-09-30T07:42:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e84138>: 
    resource group "rg-zstream-upgrade-4-21-d6bd8lqnx9k6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-mpqc9c" in resourcegroup="rg-zstream-upgrade-4-21-d6bd8lqnx9k6" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "name": "f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "status": "Failed",
      "startTime": "2026-09-30T07:56:48.455291699Z",
      "endTime": "2026-09-30T08:20:52.406066017Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofmkuvbuga9h9ge40bfgpgl5gvfp still exists (deletion dispatched at 2026-09-30T07:57:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e84138>: 
    resource group "rg-zstream-upgrade-4-21-d6bd8lqnx9k6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-mpqc9c" in resourcegroup="rg-zstream-upgrade-4-21-d6bd8lqnx9k6" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "name": "f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "status": "Failed",
      "startTime": "2026-09-30T07:56:48.455291699Z",
      "endTime": "2026-09-30T08:20:52.406066017Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofmkuvbuga9h9ge40bfgpgl5gvfp still exists (deletion dispatched at 2026-09-30T07:57:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00156e408>: 
    resource group "private-ingress-wkhlpqct8tlb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-wkhlpqct8tlb" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9cccb3e-9d83-49b8-9aa6-d3a585bd59e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9cccb3e-9d83-49b8-9aa6-d3a585bd59e7",
      "name": "a9cccb3e-9d83-49b8-9aa6-d3a585bd59e7",
      "status": "Failed",
      "startTime": "2026-09-30T07:47:33.546137807Z",
      "endTime": "2026-09-30T08:11:42.306304509Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oboi9api106r8emc4efcckh95di9 still exists (deletion dispatched at 2026-09-30T07:47:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00156e408>: 
    resource group "private-ingress-wkhlpqct8tlb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-wkhlpqct8tlb" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9cccb3e-9d83-49b8-9aa6-d3a585bd59e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9cccb3e-9d83-49b8-9aa6-d3a585bd59e7",
      "name": "a9cccb3e-9d83-49b8-9aa6-d3a585bd59e7",
      "status": "Failed",
      "startTime": "2026-09-30T07:47:33.546137807Z",
      "endTime": "2026-09-30T08:11:42.306304509Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oboi9api106r8emc4efcckh95di9 still exists (deletion dispatched at 2026-09-30T07:47:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011481e0>: 
    resource group "rg-cluster-back-version-cnrv4422rs76": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-cnrv4422rs76", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011481e0>: 
    resource group "rg-cluster-back-version-cnrv4422rs76": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-cnrv4422rs76", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00095ea98>: 
    resource group "ephemeral-osdisk-zwnb5bhrdzqz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-zwnb5bhrdzqz" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c5b0070-844b-4562-99f3-47e51a6c15ef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c5b0070-844b-4562-99f3-47e51a6c15ef",
      "name": "9c5b0070-844b-4562-99f3-47e51a6c15ef",
      "status": "Failed",
      "startTime": "2026-09-30T07:45:01.77044017Z",
      "endTime": "2026-09-30T08:09:02.54290012Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ok2d2baqjscldo4vmovim4i18pgh still exists (deletion dispatched at 2026-09-30T07:45:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00095ea98>: 
    resource group "ephemeral-osdisk-zwnb5bhrdzqz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-zwnb5bhrdzqz" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c5b0070-844b-4562-99f3-47e51a6c15ef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c5b0070-844b-4562-99f3-47e51a6c15ef",
      "name": "9c5b0070-844b-4562-99f3-47e51a6c15ef",
      "status": "Failed",
      "startTime": "2026-09-30T07:45:01.77044017Z",
      "endTime": "2026-09-30T08:09:02.54290012Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ok2d2baqjscldo4vmovim4i18pgh still exists (deletion dispatched at 2026-09-30T07:45:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001344078>: 
    resource group "rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-56tt5z" in resourcegroup="rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "name": "52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "status": "Failed",
      "startTime": "2026-09-30T07:59:56.020744554Z",
      "endTime": "2026-09-30T08:24:02.403347508Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odktt8bp0otlfbosu69ogttm3nlc still exists (deletion dispatched at 2026-09-30T07:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001344078>: 
    resource group "rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-56tt5z" in resourcegroup="rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "name": "52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "status": "Failed",
      "startTime": "2026-09-30T07:59:56.020744554Z",
      "endTime": "2026-09-30T08:24:02.403347508Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odktt8bp0otlfbosu69ogttm3nlc still exists (deletion dispatched at 2026-09-30T07:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00106a9d8>: 
    resource group "pullsecret-test-7dnj5wrv5xms": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7dnj5wrv5xms" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c782daed-b306-467e-81e3-8b6dd06ad52a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c782daed-b306-467e-81e3-8b6dd06ad52a",
      "name": "c782daed-b306-467e-81e3-8b6dd06ad52a",
      "status": "Failed",
      "startTime": "2026-09-30T07:49:52.027158436Z",
      "endTime": "2026-09-30T08:13:52.469740968Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofor9qrgni6pa0caelkcpsahuun0 still exists (deletion dispatched at 2026-09-30T07:49:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00106a9d8>: 
    resource group "pullsecret-test-7dnj5wrv5xms": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7dnj5wrv5xms" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c782daed-b306-467e-81e3-8b6dd06ad52a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c782daed-b306-467e-81e3-8b6dd06ad52a",
      "name": "c782daed-b306-467e-81e3-8b6dd06ad52a",
      "status": "Failed",
      "startTime": "2026-09-30T07:49:52.027158436Z",
      "endTime": "2026-09-30T08:13:52.469740968Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofor9qrgni6pa0caelkcpsahuun0 still exists (deletion dispatched at 2026-09-30T07:49:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001b6e0a8>: 
    resource group "customer-rg-hd4bhsrz7d7n": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-hd4bhsrz7d7n" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b56195ce-49cf-4cb1-b7c9-571ff13499d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b56195ce-49cf-4cb1-b7c9-571ff13499d0",
      "name": "b56195ce-49cf-4cb1-b7c9-571ff13499d0",
      "status": "Failed",
      "startTime": "2026-09-30T07:41:31.959909446Z",
      "endTime": "2026-09-30T08:05:32.486384816Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56onftd04005nl721veuiicq8a5s6u still exists (deletion dispatched at 2026-09-30T07:41:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001b6e0a8>: 
    resource group "customer-rg-hd4bhsrz7d7n": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-hd4bhsrz7d7n" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b56195ce-49cf-4cb1-b7c9-571ff13499d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b56195ce-49cf-4cb1-b7c9-571ff13499d0",
      "name": "b56195ce-49cf-4cb1-b7c9-571ff13499d0",
      "status": "Failed",
      "startTime": "2026-09-30T07:41:31.959909446Z",
      "endTime": "2026-09-30T08:05:32.486384816Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56onftd04005nl721veuiicq8a5s6u still exists (deletion dispatched at 2026-09-30T07:41:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001baa138>: 
    resource group "np-autoscaling-x6vzkczm54sv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-x6vzkczm54sv" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be95d509-ef21-4815-a213-b7566866b60c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be95d509-ef21-4815-a213-b7566866b60c",
      "name": "be95d509-ef21-4815-a213-b7566866b60c",
      "status": "Failed",
      "startTime": "2026-09-30T07:57:15.728931938Z",
      "endTime": "2026-09-30T08:21:22.412329675Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oig3m7l5t041s2h3535h24baigrd still exists (deletion dispatched at 2026-09-30T07:57:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001baa138>: 
    resource group "np-autoscaling-x6vzkczm54sv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-x6vzkczm54sv" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be95d509-ef21-4815-a213-b7566866b60c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be95d509-ef21-4815-a213-b7566866b60c",
      "name": "be95d509-ef21-4815-a213-b7566866b60c",
      "status": "Failed",
      "startTime": "2026-09-30T07:57:15.728931938Z",
      "endTime": "2026-09-30T08:21:22.412329675Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oig3m7l5t041s2h3535h24baigrd still exists (deletion dispatched at 2026-09-30T07:57:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002180138>: 
    resource group "feature-aggregation-7452j745q24m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-7452j745q24m" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a344fed7-604f-4718-a10f-cca3aa460a69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a344fed7-604f-4718-a10f-cca3aa460a69",
      "name": "a344fed7-604f-4718-a10f-cca3aa460a69",
      "status": "Failed",
      "startTime": "2026-09-30T08:03:38.750208141Z",
      "endTime": "2026-09-30T08:27:42.421715531Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56pklkbe3f9k9tk3gemfqinnl5654u still exists (deletion dispatched at 2026-09-30T08:03:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002180138>: 
    resource group "feature-aggregation-7452j745q24m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-7452j745q24m" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a344fed7-604f-4718-a10f-cca3aa460a69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a344fed7-604f-4718-a10f-cca3aa460a69",
      "name": "a344fed7-604f-4718-a10f-cca3aa460a69",
      "status": "Failed",
      "startTime": "2026-09-30T08:03:38.750208141Z",
      "endTime": "2026-09-30T08:27:42.421715531Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56pklkbe3f9k9tk3gemfqinnl5654u still exists (deletion dispatched at 2026-09-30T08:03:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0018c61c8>: 
    resource group "cni-cilium-5mwkmtbx9lm6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-5mwkmtbx9lm6" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc69ed49-5ecc-40ea-9a0d-84c38d022d57
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc69ed49-5ecc-40ea-9a0d-84c38d022d57",
      "name": "fc69ed49-5ecc-40ea-9a0d-84c38d022d57",
      "status": "Failed",
      "startTime": "2026-09-30T07:47:13.892958657Z",
      "endTime": "2026-09-30T08:11:22.315778755Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojjtudpbnkdj9hn5lbovlb6qujk0 still exists (deletion dispatched at 2026-09-30T07:47:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0018c61c8>: 
    resource group "cni-cilium-5mwkmtbx9lm6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-5mwkmtbx9lm6" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc69ed49-5ecc-40ea-9a0d-84c38d022d57
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc69ed49-5ecc-40ea-9a0d-84c38d022d57",
      "name": "fc69ed49-5ecc-40ea-9a0d-84c38d022d57",
      "status": "Failed",
      "startTime": "2026-09-30T07:47:13.892958657Z",
      "endTime": "2026-09-30T08:11:22.315778755Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojjtudpbnkdj9hn5lbovlb6qujk0 still exists (deletion dispatched at 2026-09-30T07:47:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0016d82b8>: 
    resource group "admin-api-serialconsole-bootdiag-bztnlzgvbtfg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-bztnlzgvbtfg" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/abff150f-53b5-4a1e-8e79-85cb6932d14c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/abff150f-53b5-4a1e-8e79-85cb6932d14c",
      "name": "abff150f-53b5-4a1e-8e79-85cb6932d14c",
      "status": "Failed",
      "startTime": "2026-09-30T07:44:47.122956765Z",
      "endTime": "2026-09-30T08:08:52.492586002Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oclgsj9oapvabnc01b65dqdln9hs still exists (deletion dispatched at 2026-09-30T07:44:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0016d82b8>: 
    resource group "admin-api-serialconsole-bootdiag-bztnlzgvbtfg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-bztnlzgvbtfg" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/abff150f-53b5-4a1e-8e79-85cb6932d14c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/abff150f-53b5-4a1e-8e79-85cb6932d14c",
      "name": "abff150f-53b5-4a1e-8e79-85cb6932d14c",
      "status": "Failed",
      "startTime": "2026-09-30T07:44:47.122956765Z",
      "endTime": "2026-09-30T08:08:52.492586002Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56oclgsj9oapvabnc01b65dqdln9hs still exists (deletion dispatched at 2026-09-30T07:44:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f6c198>: 
    resource group "private-kas-htntrcf5948s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-htntrcf5948s" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e211cf-72b5-460c-86b9-42f8c0ab026d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e211cf-72b5-460c-86b9-42f8c0ab026d",
      "name": "f1e211cf-72b5-460c-86b9-42f8c0ab026d",
      "status": "Failed",
      "startTime": "2026-09-30T07:53:38.631153455Z",
      "endTime": "2026-09-30T08:17:42.452924542Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofmlsp87gpoiuva34g4ukacv58an still exists (deletion dispatched at 2026-09-30T07:53:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f6c198>: 
    resource group "private-kas-htntrcf5948s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-htntrcf5948s" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e211cf-72b5-460c-86b9-42f8c0ab026d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e211cf-72b5-460c-86b9-42f8c0ab026d",
      "name": "f1e211cf-72b5-460c-86b9-42f8c0ab026d",
      "status": "Failed",
      "startTime": "2026-09-30T07:53:38.631153455Z",
      "endTime": "2026-09-30T08:17:42.452924542Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofmlsp87gpoiuva34g4ukacv58an still exists (deletion dispatched at 2026-09-30T07:53:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001190090>: 
    resource group "rg-zstream-upgrade-5-0-vztfjbhwj86s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-dbc7n2" in resourcegroup="rg-zstream-upgrade-5-0-vztfjbhwj86s" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85",
      "name": "86d21c47-4deb-485c-adc0-8dba2db04e85",
      "status": "Failed",
      "startTime": "2026-09-30T08:00:45.311282131Z",
      "endTime": "2026-09-30T08:24:52.368269792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojo9ccai43f073bp4dpipm3b14ma still exists (deletion dispatched at 2026-09-30T08:01:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001190090>: 
    resource group "rg-zstream-upgrade-5-0-vztfjbhwj86s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-dbc7n2" in resourcegroup="rg-zstream-upgrade-5-0-vztfjbhwj86s" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85",
      "name": "86d21c47-4deb-485c-adc0-8dba2db04e85",
      "status": "Failed",
      "startTime": "2026-09-30T08:00:45.311282131Z",
      "endTime": "2026-09-30T08:24:52.368269792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojo9ccai43f073bp4dpipm3b14ma still exists (deletion dispatched at 2026-09-30T08:01:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:409 +0x87 fp=0xc0000d9e90 sp=0xc0000d9e20 pc=0xbff15a7
github.com/Azure/ARO-HCP/test/util/framework.runResourceGroupCleanup.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:410 +0x7e fp=0xc0000d9f78 sp=0xc0000d9e90 pc=0xbff12be
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000d9fe0 sp=0xc0000d9f78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3248
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

goroutine 3283 gp=0xc0011bfdc0 m=nil [select, 2 minutes]:
runtime.gopark(0xc000e65f48?, 0x2?, 0xc0?, 0x5d?, 0xc000e65ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000df4d78 sp=0xc000df4d58 pc=0x54317ae
runtime.selectgo(0xc000df4f48, 0xc000e65ef0, 0xc00191c0c0?, 0x0, 0xc0028017d0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000df4eb8 sp=0xc000df4d78 pc=0x540c3b7
net/http.(*persistConn).writeLoop(0xc0013e3b00)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000df4fc8 sp=0xc000df4eb8 pc=0x5803227
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000df4fe0 sp=0xc000df4fc8 pc=0x57ffec5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000df4fe8 sp=0xc000df4fe0 pc=0x5439fc1
created by net/http.(*Transport).dialConn in goroutine 3249
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 3232 gp=0xc0002616c0 m=nil [select]:
runtime.gopark(0xc001a7f768?, 0x2?, 0x40?, 0x0?, 0xc001a7f75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001a7f5e8 sp=0xc001a7f5c8 pc=0x54317ae
runtime.selectgo(0xc001a7f768, 0xc001a7f758, 0xc001a7f788?, 0x0, 0xf83b8ce?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001a7f728 sp=0xc001a7f5e8 pc=0x540c3b7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x108bfe40, 0xc00066ad20}, 0xf7fc714?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/internal/shared/shared.go:36 +0x76 fp=0xc001a7f798 sp=0xc001a7f728 pc=0x5c1c996
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x10919ee0, {0x108bfe40, 0xc00066ad20}, 0x542ed93)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/runtime/poller.go:284 +0x732 fp=0xc001a7fa98 sp=0xc001a7f798 pc=0x5e69a92
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x108bfe40, 0xc00066ad20}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:591 +0x365 fp=0xc001a7fc20 sp=0xc001a7fa98 pc=0xbfaf265
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x108bfe40?, 0xc00066ad20?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:628 +0x72 fp=0xc001a7fcc8 sp=0xc001a7fc20 pc=0xbfafa12
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x108bfe40?, 0xc00066ad20?}, 0xc00066ad30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc001a7fd20 sp=0xc001a7fcc8 pc=0x5bd1a22
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x108bfe40, 0xc00066ad20}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc001a7fe68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc001a7fd98 sp=0xc001a7fd20 pc=0x5bcface
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x108bfdd0, 0xc000f19220}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15}, 0x15d3ef79800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:626 +0x285 fp=0xc001a7fef0 sp=0xc001a7fd98 pc=0xbfaf785
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:799 +0x7e fp=0xc001a7ff78 sp=0xc001a7fef0 pc=0xbfb219e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc001a7ffe0 sp=0xc001a7ff78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001a7ffe8 sp=0xc001a7ffe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3273
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x543bde3
rdx    0x0
rdi    0x169679d0
rsi    0x80
rbp    0x7ffc297c0df0
rsp    0x7ffc297c0da8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0x16965440
r15    0x5
rip    0x543bde1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:409 +0x87 fp=0xc0000d9e90 sp=0xc0000d9e20 pc=0xbff15a7
github.com/Azure/ARO-HCP/test/util/framework.runResourceGroupCleanup.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:410 +0x7e fp=0xc0000d9f78 sp=0xc0000d9e90 pc=0xbff12be
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000d9fe0 sp=0xc0000d9f78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3248
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

goroutine 3283 gp=0xc0011bfdc0 m=nil [select, 2 minutes]:
runtime.gopark(0xc000e65f48?, 0x2?, 0xc0?, 0x5d?, 0xc000e65ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000df4d78 sp=0xc000df4d58 pc=0x54317ae
runtime.selectgo(0xc000df4f48, 0xc000e65ef0, 0xc00191c0c0?, 0x0, 0xc0028017d0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000df4eb8 sp=0xc000df4d78 pc=0x540c3b7
net/http.(*persistConn).writeLoop(0xc0013e3b00)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000df4fc8 sp=0xc000df4eb8 pc=0x5803227
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000df4fe0 sp=0xc000df4fc8 pc=0x57ffec5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000df4fe8 sp=0xc000df4fe0 pc=0x5439fc1
created by net/http.(*Transport).dialConn in goroutine 3249
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 3232 gp=0xc0002616c0 m=nil [select]:
runtime.gopark(0xc001a7f768?, 0x2?, 0x40?, 0x0?, 0xc001a7f75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001a7f5e8 sp=0xc001a7f5c8 pc=0x54317ae
runtime.selectgo(0xc001a7f768, 0xc001a7f758, 0xc001a7f788?, 0x0, 0xf83b8ce?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001a7f728 sp=0xc001a7f5e8 pc=0x540c3b7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x108bfe40, 0xc00066ad20}, 0xf7fc714?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/internal/shared/shared.go:36 +0x76 fp=0xc001a7f798 sp=0xc001a7f728 pc=0x5c1c996
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x10919ee0, {0x108bfe40, 0xc00066ad20}, 0x542ed93)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/runtime/poller.go:284 +0x732 fp=0xc001a7fa98 sp=0xc001a7f798 pc=0x5e69a92
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x108bfe40, 0xc00066ad20}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:591 +0x365 fp=0xc001a7fc20 sp=0xc001a7fa98 pc=0xbfaf265
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x108bfe40?, 0xc00066ad20?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:628 +0x72 fp=0xc001a7fcc8 sp=0xc001a7fc20 pc=0xbfafa12
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x108bfe40?, 0xc00066ad20?}, 0xc00066ad30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc001a7fd20 sp=0xc001a7fcc8 pc=0x5bd1a22
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x108bfe40, 0xc00066ad20}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc001a7fe68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc001a7fd98 sp=0xc001a7fd20 pc=0x5bcface
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x108bfdd0, 0xc000f19220}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15}, 0x15d3ef79800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:626 +0x285 fp=0xc001a7fef0 sp=0xc001a7fd98 pc=0xbfaf785
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:799 +0x7e fp=0xc001a7ff78 sp=0xc001a7fef0 pc=0xbfb219e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc001a7ffe0 sp=0xc001a7ff78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001a7ffe8 sp=0xc001a7ffe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3273
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x543bde3
rdx    0x0
rdi    0x169679d0
rsi    0x80
rbp    0x7ffc297c0df0
rsp    0x7ffc297c0da8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0x16965440
r15    0x5
rip    0x543bde1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011781c8>: 
    resource group "idms-6clt25jfd748": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6clt25jfd748" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95e22e88-1144-4d00-8cc9-dc9f1a6c3ffb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95e22e88-1144-4d00-8cc9-dc9f1a6c3ffb",
      "name": "95e22e88-1144-4d00-8cc9-dc9f1a6c3ffb",
      "status": "Failed",
      "startTime": "2026-09-30T07:43:50.391649068Z",
      "endTime": "2026-09-30T08:07:52.519744698Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofblporgmut8brildeurh8bqh0jm still exists (deletion dispatched at 2026-09-30T07:44:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011781c8>: 
    resource group "idms-6clt25jfd748": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6clt25jfd748" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95e22e88-1144-4d00-8cc9-dc9f1a6c3ffb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95e22e88-1144-4d00-8cc9-dc9f1a6c3ffb",
      "name": "95e22e88-1144-4d00-8cc9-dc9f1a6c3ffb",
      "status": "Failed",
      "startTime": "2026-09-30T07:43:50.391649068Z",
      "endTime": "2026-09-30T08:07:52.519744698Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofblporgmut8brildeurh8bqh0jm still exists (deletion dispatched at 2026-09-30T07:44:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001702090>: 
    resource group "fips-enabled-xs9fxbdgnwhz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-xs9fxbdgnwhz" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278",
      "name": "81d3effe-1da4-43c5-8596-8a54252ca278",
      "status": "Failed",
      "startTime": "2026-09-30T07:45:23.642226584Z",
      "endTime": "2026-09-30T08:09:32.307764104Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obh88cddik6iio0cu4ci35j02k7f still exists (deletion dispatched at 2026-09-30T07:45:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001702090>: 
    resource group "fips-enabled-xs9fxbdgnwhz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-xs9fxbdgnwhz" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278",
      "name": "81d3effe-1da4-43c5-8596-8a54252ca278",
      "status": "Failed",
      "startTime": "2026-09-30T07:45:23.642226584Z",
      "endTime": "2026-09-30T08:09:32.307764104Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obh88cddik6iio0cu4ci35j02k7f still exists (deletion dispatched at 2026-09-30T07:45:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000d94390>: 
    resource group "rg-candidate-4-22-7rr572k8p4nx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hqg9nr" in resourcegroup="rg-candidate-4-22-7rr572k8p4nx" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5146bdc-a88e-43b4-ade8-4264fc68118b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5146bdc-a88e-43b4-ade8-4264fc68118b",
      "name": "c5146bdc-a88e-43b4-ade8-4264fc68118b",
      "status": "Failed",
      "startTime": "2026-09-30T07:52:17.26155411Z",
      "endTime": "2026-09-30T08:16:22.313143273Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojq845tseueb8p9jb82rbbpq2mbb still exists (deletion dispatched at 2026-09-30T07:52:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000d94390>: 
    resource group "rg-candidate-4-22-7rr572k8p4nx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hqg9nr" in resourcegroup="rg-candidate-4-22-7rr572k8p4nx" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5146bdc-a88e-43b4-ade8-4264fc68118b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5146bdc-a88e-43b4-ade8-4264fc68118b",
      "name": "c5146bdc-a88e-43b4-ade8-4264fc68118b",
      "status": "Failed",
      "startTime": "2026-09-30T07:52:17.26155411Z",
      "endTime": "2026-09-30T08:16:22.313143273Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojq845tseueb8p9jb82rbbpq2mbb still exists (deletion dispatched at 2026-09-30T07:52:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f38108>: 
    resource group "v20260901preview-fphhhngljlhp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-fphhhngljlhp" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc213230-cecc-45fc-a51e-9966fc3f25a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc213230-cecc-45fc-a51e-9966fc3f25a4",
      "name": "dc213230-cecc-45fc-a51e-9966fc3f25a4",
      "status": "Failed",
      "startTime": "2026-09-30T07:49:49.418893525Z",
      "endTime": "2026-09-30T08:13:52.389241283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ocsot5cpua59vgr84hkknkr96gf6 still exists (deletion dispatched at 2026-09-30T07:49:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f38108>: 
    resource group "v20260901preview-fphhhngljlhp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-fphhhngljlhp" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc213230-cecc-45fc-a51e-9966fc3f25a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc213230-cecc-45fc-a51e-9966fc3f25a4",
      "name": "dc213230-cecc-45fc-a51e-9966fc3f25a4",
      "status": "Failed",
      "startTime": "2026-09-30T07:49:49.418893525Z",
      "endTime": "2026-09-30T08:13:52.389241283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ocsot5cpua59vgr84hkknkr96gf6 still exists (deletion dispatched at 2026-09-30T07:49:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001396090>: 
    resource group "rg-cp-ystream-upgrade-4-23-wm46s5gt65dl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-23-pg8svf" in resourcegroup="rg-cp-ystream-upgrade-4-23-wm46s5gt65dl" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4f33f8f6-72a9-44ff-8c84-6b45107fd2f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4f33f8f6-72a9-44ff-8c84-6b45107fd2f9",
      "name": "4f33f8f6-72a9-44ff-8c84-6b45107fd2f9",
      "status": "Failed",
      "startTime": "2026-09-30T07:57:46.367143777Z",
      "endTime": "2026-09-30T08:21:52.387303059Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odfbmlrdpspcs2jrvj7ok5eocb47 still exists (deletion dispatched at 2026-09-30T07:57:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001396090>: 
    resource group "rg-cp-ystream-upgrade-4-23-wm46s5gt65dl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-23-pg8svf" in resourcegroup="rg-cp-ystream-upgrade-4-23-wm46s5gt65dl" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4f33f8f6-72a9-44ff-8c84-6b45107fd2f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4f33f8f6-72a9-44ff-8c84-6b45107fd2f9",
      "name": "4f33f8f6-72a9-44ff-8c84-6b45107fd2f9",
      "status": "Failed",
      "startTime": "2026-09-30T07:57:46.367143777Z",
      "endTime": "2026-09-30T08:21:52.387303059Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odfbmlrdpspcs2jrvj7ok5eocb47 still exists (deletion dispatched at 2026-09-30T07:57:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:301]: failed to update cluster with new KMS key with the second rotation
Unexpected error:
    <*fmt.wrapErrors | 0xc000dfb3e0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-vxcvln6bjmck" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-vxcvln6bjmck, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:301]: failed to update cluster with new KMS key with the second rotation
Unexpected error:
    <*fmt.wrapErrors | 0xc000dfb3e0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-vxcvln6bjmck" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-vxcvln6bjmck, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019ec0d8>: 
    resource group "rg-zstream-upgrade-4-23-gq8tt4j82btd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-23-jjk68g" in resourcegroup="rg-zstream-upgrade-4-23-gq8tt4j82btd" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/259e7251-93cc-4a54-a228-5ab61010d78f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/259e7251-93cc-4a54-a228-5ab61010d78f",
      "name": "259e7251-93cc-4a54-a228-5ab61010d78f",
      "status": "Failed",
      "startTime": "2026-09-30T08:02:15.27366157Z",
      "endTime": "2026-09-30T08:26:22.388447131Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ogva7br99jqa1217jecnq155d1ej still exists (deletion dispatched at 2026-09-30T08:02:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019ec0d8>: 
    resource group "rg-zstream-upgrade-4-23-gq8tt4j82btd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-23-jjk68g" in resourcegroup="rg-zstream-upgrade-4-23-gq8tt4j82btd" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/259e7251-93cc-4a54-a228-5ab61010d78f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/259e7251-93cc-4a54-a228-5ab61010d78f",
      "name": "259e7251-93cc-4a54-a228-5ab61010d78f",
      "status": "Failed",
      "startTime": "2026-09-30T08:02:15.27366157Z",
      "endTime": "2026-09-30T08:26:22.388447131Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ogva7br99jqa1217jecnq155d1ej still exists (deletion dispatched at 2026-09-30T08:02:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010642a0>: 
    resource group "external-auth-cluster-4nlpdcdjkq8m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-4nlpdcdjkq8m" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f11d658-22eb-436f-a6d0-266fcfe0ad04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f11d658-22eb-436f-a6d0-266fcfe0ad04",
      "name": "0f11d658-22eb-436f-a6d0-266fcfe0ad04",
      "status": "Failed",
      "startTime": "2026-09-30T07:50:30.578221498Z",
      "endTime": "2026-09-30T08:14:32.551976542Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odm3uljbr1bc72mqo3pd7c5dcrar still exists (deletion dispatched at 2026-09-30T07:50:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010642a0>: 
    resource group "external-auth-cluster-4nlpdcdjkq8m": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-4nlpdcdjkq8m" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f11d658-22eb-436f-a6d0-266fcfe0ad04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f11d658-22eb-436f-a6d0-266fcfe0ad04",
      "name": "0f11d658-22eb-436f-a6d0-266fcfe0ad04",
      "status": "Failed",
      "startTime": "2026-09-30T07:50:30.578221498Z",
      "endTime": "2026-09-30T08:14:32.551976542Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odm3uljbr1bc72mqo3pd7c5dcrar still exists (deletion dispatched at 2026-09-30T07:50:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000cd63f0>: 
    resource group "arm64-vm-cluster-62xw9fcdx42j": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-62xw9fcdx42j" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d160b43-5f34-4632-80cb-5de8f2fd3f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d160b43-5f34-4632-80cb-5de8f2fd3f41",
      "name": "7d160b43-5f34-4632-80cb-5de8f2fd3f41",
      "status": "Failed",
      "startTime": "2026-09-30T07:46:16.079175081Z",
      "endTime": "2026-09-30T08:10:22.378932829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ocn9um98ldcojp1pl53iclpb0kl7 still exists (deletion dispatched at 2026-09-30T07:46:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000cd63f0>: 
    resource group "arm64-vm-cluster-62xw9fcdx42j": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-62xw9fcdx42j" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d160b43-5f34-4632-80cb-5de8f2fd3f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d160b43-5f34-4632-80cb-5de8f2fd3f41",
      "name": "7d160b43-5f34-4632-80cb-5de8f2fd3f41",
      "status": "Failed",
      "startTime": "2026-09-30T07:46:16.079175081Z",
      "endTime": "2026-09-30T08:10:22.378932829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ocn9um98ldcojp1pl53iclpb0kl7 still exists (deletion dispatched at 2026-09-30T07:46:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012cc090>: 
    resource group "ea-list-tcgjz7swjvdw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listrk7c29" in resourcegroup="ea-list-tcgjz7swjvdw" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19fe26bb-4a1e-4ebc-baa6-f57c53e83296
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19fe26bb-4a1e-4ebc-baa6-f57c53e83296",
      "name": "19fe26bb-4a1e-4ebc-baa6-f57c53e83296",
      "status": "Failed",
      "startTime": "2026-09-30T07:43:53.484653514Z",
      "endTime": "2026-09-30T08:08:02.489234786Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56p42at8ondug5mpb3mmh6f9hh71ck still exists (deletion dispatched at 2026-09-30T07:44:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012cc090>: 
    resource group "ea-list-tcgjz7swjvdw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listrk7c29" in resourcegroup="ea-list-tcgjz7swjvdw" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19fe26bb-4a1e-4ebc-baa6-f57c53e83296
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19fe26bb-4a1e-4ebc-baa6-f57c53e83296",
      "name": "19fe26bb-4a1e-4ebc-baa6-f57c53e83296",
      "status": "Failed",
      "startTime": "2026-09-30T07:43:53.484653514Z",
      "endTime": "2026-09-30T08:08:02.489234786Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56p42at8ondug5mpb3mmh6f9hh71ck still exists (deletion dispatched at 2026-09-30T07:44:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0005468d0>: 
    resource group "rg-candidate-4-20-6zgdjl96fd2x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-dn6n8z" in resourcegroup="rg-candidate-4-20-6zgdjl96fd2x" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3ec571ff-78ed-435f-b6c6-33518aa00501
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3ec571ff-78ed-435f-b6c6-33518aa00501",
      "name": "3ec571ff-78ed-435f-b6c6-33518aa00501",
      "status": "Failed",
      "startTime": "2026-09-30T07:55:49.208034763Z",
      "endTime": "2026-09-30T08:19:52.415025374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojtndhilnoii39q1v1t13gbjb4ah still exists (deletion dispatched at 2026-09-30T07:55:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0005468d0>: 
    resource group "rg-candidate-4-20-6zgdjl96fd2x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-dn6n8z" in resourcegroup="rg-candidate-4-20-6zgdjl96fd2x" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3ec571ff-78ed-435f-b6c6-33518aa00501
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3ec571ff-78ed-435f-b6c6-33518aa00501",
      "name": "3ec571ff-78ed-435f-b6c6-33518aa00501",
      "status": "Failed",
      "startTime": "2026-09-30T07:55:49.208034763Z",
      "endTime": "2026-09-30T08:19:52.415025374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojtndhilnoii39q1v1t13gbjb4ah still exists (deletion dispatched at 2026-09-30T07:55:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001ed0138>: 
    resource group "rg-hs-pre--7p7rhfskdkg8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-mxbwvc" in resourcegroup="rg-hs-pre--7p7rhfskdkg8" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/38b5c072-1d7b-4021-9dde-2bef3cf5ae9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/38b5c072-1d7b-4021-9dde-2bef3cf5ae9a",
      "name": "38b5c072-1d7b-4021-9dde-2bef3cf5ae9a",
      "status": "Failed",
      "startTime": "2026-09-30T07:49:12.08839353Z",
      "endTime": "2026-09-30T08:13:12.35854499Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obncvdr9t1vsvq3snfkacnvh3cke still exists (deletion dispatched at 2026-09-30T07:49:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001ed0138>: 
    resource group "rg-hs-pre--7p7rhfskdkg8": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-mxbwvc" in resourcegroup="rg-hs-pre--7p7rhfskdkg8" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/38b5c072-1d7b-4021-9dde-2bef3cf5ae9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/38b5c072-1d7b-4021-9dde-2bef3cf5ae9a",
      "name": "38b5c072-1d7b-4021-9dde-2bef3cf5ae9a",
      "status": "Failed",
      "startTime": "2026-09-30T07:49:12.08839353Z",
      "endTime": "2026-09-30T08:13:12.35854499Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obncvdr9t1vsvq3snfkacnvh3cke still exists (deletion dispatched at 2026-09-30T07:49:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T09:55:42Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0283264-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-84tpd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-84tpd (namespace arobit) on cluster ci01-j0283264-mgmt-2 has averaged 123.4% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T09:10:27Z
  Ended:       2026-09-30T10:12:28Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t56ofmlsp87gpoiuva34g4ukacv58an-private-kas", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t56ofmlsp87gpoiuva34g4ukacv58an-private-kas (management cluster ci01-j0283264-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.

alert fired 16 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:36:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:36:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:36:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:36:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:36:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:36:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:44:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:44:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:44:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:44:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:44:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:46:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-30T09:10:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-30T09:10:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-30T09:10:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-30T09:10:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mqkvs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T09:05:33Z
  Ended:       2026-09-30T09:25:32Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0283264-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.44:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/services-j0283264", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8bc58bd6b-kmgbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="kubernetesevents"
  Description: Kusto log data for table kubernetesevents on cluster ci01-j0283264-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Resolved
  Started:     2026-09-30T09:06:32Z
  Ended:       2026-09-30T09:26:32Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0283264-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.44:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/services-j0283264", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8bc58bd6b-kmgbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="kubernetesevents"
  Description: Kusto log data for table kubernetesevents on cluster ci01-j0283264-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 32 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="cloud-network-config-controller-6b7699b59d-mtvjg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1aa0eaea-3879-4a75-b74f-75ef531c3b1d"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/cloud-network-config-controller-6b7699b59d-mtvjg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="cloud-network-config-controller-c9f8f4988-98sd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1b3a2719-6e7a-4a5b-aeae-d318e5e4d17e"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/cloud-network-config-controller-c9f8f4988-98sd7 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="csi-snapshot-controller-ff4fdf4ff-8fn8s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="523581cf-5012-43ac-a402-c7a0a8acaac4"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/csi-snapshot-controller-ff4fdf4ff-8fn8s (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="cloud-network-config-controller-5c94ccc4b5-cg2cp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dafd5d14-e303-47e0-b4e1-d01b00ea97f1"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/cloud-network-config-controller-5c94ccc4b5-cg2cp (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="cloud-network-config-controller-6b7699b59d-mtvjg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1aa0eaea-3879-4a75-b74f-75ef531c3b1d"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/cloud-network-config-controller-6b7699b59d-mtvjg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="csi-snapshot-controller-7477f9489c-8hwvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a334069-0f8a-44c9-ab3a-5524390a74c8"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/csi-snapshot-controller-7477f9489c-8hwvb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="cloud-network-config-controller-749db5479f-29nst", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ad67988d-3f68-4e02-84b2-957b828c23ea"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/cloud-network-config-controller-749db5479f-29nst (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="cloud-network-config-controller-79d7c84c4-jpmxx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="350fe4f2-fba9-41ad-825a-5166b3b5b8f2"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/cloud-network-config-controller-79d7c84c4-jpmxx (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="csi-snapshot-controller-5cc4dd7c9d-2tkdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4766f6e2-d1bc-40e5-9418-ada111a189ef"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/csi-snapshot-controller-5cc4dd7c9d-2tkdw (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="csi-snapshot-controller-6445b45cbd-g46qp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4c46f8cc-e7a5-456d-88e9-4d97f220cbc0"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/csi-snapshot-controller-6445b45cbd-g46qp (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="csi-snapshot-controller-68f64f94cf-f2r4v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e4f7cb9e-9e38-408b-b971-4b23a9e6fc0e"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/csi-snapshot-controller-68f64f94cf-f2r4v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="cloud-network-config-controller-c9f8f4988-98sd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1b3a2719-6e7a-4a5b-aeae-d318e5e4d17e"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/cloud-network-config-controller-c9f8f4988-98sd7 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="csi-snapshot-controller-7cf66cc49d-fnzp4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6f42eca0-0eb8-412d-9fbb-e74e80100964"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/csi-snapshot-controller-7cf66cc49d-fnzp4 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="csi-snapshot-controller-6445b45cbd-g46qp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4c46f8cc-e7a5-456d-88e9-4d97f220cbc0"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/csi-snapshot-controller-6445b45cbd-g46qp (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="csi-snapshot-controller-ff4fdf4ff-8fn8s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="523581cf-5012-43ac-a402-c7a0a8acaac4"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/csi-snapshot-controller-ff4fdf4ff-8fn8s (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="cloud-network-config-controller-85b777dd49-s25s6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="7f4bec03-7c59-4bee-bd1a-b303f9184e5c"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/cloud-network-config-controller-85b777dd49-s25s6 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="cloud-network-config-controller-79d7c84c4-jpmxx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="350fe4f2-fba9-41ad-825a-5166b3b5b8f2"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/cloud-network-config-controller-79d7c84c4-jpmxx (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="csi-snapshot-controller-7cf66cc49d-fnzp4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6f42eca0-0eb8-412d-9fbb-e74e80100964"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/csi-snapshot-controller-7cf66cc49d-fnzp4 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="cloud-network-config-controller-85b777dd49-s25s6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="7f4bec03-7c59-4bee-bd1a-b303f9184e5c"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/cloud-network-config-controller-85b777dd49-s25s6 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="cloud-network-config-controller-c48d57456-gkqfz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="162d1556-afda-4dc1-97da-7279eda02fab"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/cloud-network-config-controller-c48d57456-gkqfz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="cloud-network-config-controller-749db5479f-29nst", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ad67988d-3f68-4e02-84b2-957b828c23ea"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/cloud-network-config-controller-749db5479f-29nst (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="cloud-network-config-controller-5687d9df7d-mc789", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c80807d8-f734-41fa-bf72-fa4544a1127f"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/cloud-network-config-controller-5687d9df7d-mc789 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="csi-snapshot-controller-546555d6dd-whrqm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6e800052-3ed4-4832-ac6e-ba3a34c7eda1"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/csi-snapshot-controller-546555d6dd-whrqm (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="csi-snapshot-controller-68698d756-2sjcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ea8389c4-6821-4c53-af9f-460b56d932d9"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/csi-snapshot-controller-68698d756-2sjcb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="cloud-network-config-controller-c48d57456-gkqfz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="162d1556-afda-4dc1-97da-7279eda02fab"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/cloud-network-config-controller-c48d57456-gkqfz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="csi-snapshot-controller-5cc4dd7c9d-2tkdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4766f6e2-d1bc-40e5-9418-ada111a189ef"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/csi-snapshot-controller-5cc4dd7c9d-2tkdw (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="csi-snapshot-controller-68698d756-2sjcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ea8389c4-6821-4c53-af9f-460b56d932d9"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/csi-snapshot-controller-68698d756-2sjcb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="csi-snapshot-controller-68f64f94cf-f2r4v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e4f7cb9e-9e38-408b-b971-4b23a9e6fc0e"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/csi-snapshot-controller-68f64f94cf-f2r4v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="csi-snapshot-controller-546555d6dd-whrqm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6e800052-3ed4-4832-ac6e-ba3a34c7eda1"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/csi-snapshot-controller-546555d6dd-whrqm (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="csi-snapshot-controller-7477f9489c-8hwvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a334069-0f8a-44c9-ab3a-5524390a74c8"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/csi-snapshot-controller-7477f9489c-8hwvb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="cloud-network-config-controller-5c94ccc4b5-cg2cp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dafd5d14-e303-47e0-b4e1-d01b00ea97f1"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/cloud-network-config-controller-5c94ccc4b5-cg2cp (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="cloud-network-config-controller-5687d9df7d-mc789", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c80807d8-f734-41fa-bf72-fa4544a1127f"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/cloud-network-config-controller-5687d9df7d-mc789 (controller) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:487]: ClusterVersion had no completed update entry in history
Expected
    <string>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:487]: ClusterVersion had no completed update entry in history
Expected
    <string>: ...
not to be empty

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:42:08Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:42:08Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T06:49:03Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T06:49:04Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:07:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-czf5c", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-czf5c has been in a non-ready state for longer than 5 minutes.

alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:32:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:32:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T06:39:00Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T06:39:00Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:12:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:12:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:11:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:11:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-78dt7bb9qdx9" to finish creating: GET https://rp.j9082368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "name": "e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "status": "Failed",
  "startTime": "2026-09-30T05:49:26.691969203Z",
  "endTime": "2026-09-30T06:08:29.964580175Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-78dt7bb9qdx9" to finish creating: GET https://rp.j9082368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "name": "e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "status": "Failed",
  "startTime": "2026-09-30T05:49:26.691969203Z",
  "endTime": "2026-09-30T06:08:29.964580175Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0010db2a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-gwhcb7q77s5g" to finish creating: GET https://rp.j1931648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26eb862-dd0d-4748-bf3d-aaa27c84e026
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26eb862-dd0d-4748-bf3d-aaa27c84e026",
      "name": "e26eb862-dd0d-4748-bf3d-aaa27c84e026",
      "status": "Failed",
      "startTime": "2026-09-30T04:22:37.324871209Z",
      "endTime": "2026-09-30T04:41:46.747553014Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gwhcb7q77s5g--managed/providers/Microsoft.Compute/virtualMachines/l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gwhcb7q77s5g--managed/providers/Microsoft.Compute/virtualMachines/l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0010db2a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-gwhcb7q77s5g" to finish creating: GET https://rp.j1931648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26eb862-dd0d-4748-bf3d-aaa27c84e026
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e26eb862-dd0d-4748-bf3d-aaa27c84e026",
      "name": "e26eb862-dd0d-4748-bf3d-aaa27c84e026",
      "status": "Failed",
      "startTime": "2026-09-30T04:22:37.324871209Z",
      "endTime": "2026-09-30T04:41:46.747553014Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gwhcb7q77s5g--managed/providers/Microsoft.Compute/virtualMachines/l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gwhcb7q77s5g--managed/providers/Microsoft.Compute/virtualMachines/l4b9n5d1w5j2a3f-autoscale-noaz-wmbj7-vtxcx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:130]: failed to create customer resources for aggregated feature cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009224e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-agg-cluster-kv5cgd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-agg-cluster-kv5cgd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-agg-cluster-kv5cgd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:130]: failed to create customer resources for aggregated feature cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009224e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-agg-cluster-kv5cgd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-agg-cluster-kv5cgd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-agg-cluster-kv5cgd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-w4nsqz" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000d2e780>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nhrfp5qbdv2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-w4nsqz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-w4nsqz" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000d2e780>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nhrfp5qbdv2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-w4nsqz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-lr7n9c" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00144e180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-zh65jflwpg9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-lr7n9c
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-lr7n9c" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00144e180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-zh65jflwpg9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-lr7n9c
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-nkn9cp" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000fb60c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2s89ppfpwxxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-nkn9cp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-nkn9cp" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000fb60c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2s89ppfpwxxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-nkn9cp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-8g7jd8" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014b4e10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-c244qsxrhnbn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-8g7jd8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-8g7jd8" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014b4e10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-c244qsxrhnbn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-8g7jd8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d6f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-t8kp5gffh7ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e",
      "name": "fbb7c321-b5c6-4145-9be8-48a64282647e",
      "status": "Failed",
      "startTime": "2026-09-30T22:47:02.188860165Z",
      "endTime": "2026-09-30T23:06:03.834732494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d6f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-t8kp5gffh7ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e",
      "name": "fbb7c321-b5c6-4145-9be8-48a64282647e",
      "status": "Failed",
      "startTime": "2026-09-30T22:47:02.188860165Z",
      "endTime": "2026-09-30T23:06:03.834732494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-52n697" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0015481e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-52jx5pj75l4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-52n697
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-52n697" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0015481e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-52jx5pj75l4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-52n697
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-j4z2m9" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000c73320>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-xlhwbtlldbgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-j4z2m9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-j4z2m9" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000c73320>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-xlhwbtlldbgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-j4z2m9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-b9ld4k" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000a93bf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-m7g7x4rhlkmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-b9ld4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-b9ld4k" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000a93bf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-m7g7x4rhlkmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-b9ld4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-6czr45" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00106e2a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zs8jz58nhzb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-6czr45
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-6czr45" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00106e2a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zs8jz58nhzb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-6czr45
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d9vb4z" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000f5c990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-v5vn6xvq7g9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-d9vb4z
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d9vb4z" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000f5c990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-v5vn6xvq7g9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-d9vb4z
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xw9s99" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00095f590>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-248rf82xp8ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xw9s99
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xw9s99" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00095f590>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-248rf82xp8ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xw9s99
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc001088480>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-mmgvqz" in resourcegroup="rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846",
      "name": "b1caf488-0088-4fa0-8a37-c1f30126d846",
      "status": "Failed",
      "startTime": "2026-09-30T20:37:05.655658972Z",
      "endTime": "2026-09-30T20:56:13.566012243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc001088480>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-mmgvqz" in resourcegroup="rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846",
      "name": "b1caf488-0088-4fa0-8a37-c1f30126d846",
      "status": "Failed",
      "startTime": "2026-09-30T20:37:05.655658972Z",
      "endTime": "2026-09-30T20:56:13.566012243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-20-t74pss"
Unexpected error:
    <*fmt.wrapError | 0xc000b40260>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-t74pss-7wvjkj" in resourcegroup="rg-candidate-4-20-bmfs7fmklpfb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-bmfs7fmklpfb/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-t74pss-7wvjkj/operationStatuses/08584108180410013833
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-cr5ick5opsuzu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-20-t74pss"
Unexpected error:
    <*fmt.wrapError | 0xc000b40260>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-t74pss-7wvjkj" in resourcegroup="rg-candidate-4-20-bmfs7fmklpfb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-bmfs7fmklpfb/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-t74pss-7wvjkj/operationStatuses/08584108180410013833
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-cr5ick5opsuzu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013120e0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-sttxg5n27hh5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc",
      "name": "95422086-7eb5-4fab-930f-f78b5c9185bc",
      "status": "Failed",
      "startTime": "2026-10-01T00:05:50.759671414Z",
      "endTime": "2026-10-01T00:24:59.018405341Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013120e0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-sttxg5n27hh5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc",
      "name": "95422086-7eb5-4fab-930f-f78b5c9185bc",
      "status": "Failed",
      "startTime": "2026-10-01T00:05:50.759671414Z",
      "endTime": "2026-10-01T00:24:59.018405341Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000d541a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-mv9rpr" in resourcegroup="rg-np-skip-minor-mv9rpr-lccf5r2wcvcf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mv9rpr-lccf5r2wcvcf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mv9rpr/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000d541a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-mv9rpr" in resourcegroup="rg-np-skip-minor-mv9rpr-lccf5r2wcvcf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mv9rpr-lccf5r2wcvcf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mv9rpr/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-5-0-b96xbr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc000e76f48>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-5-0-b96xbr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc000e76f48>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc0009b8bc0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-sgrs2v" in resourcegroup="rg-np-version-upgrade-sgrs2v-l46fqcfczcg5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-sgrs2v-l46fqcfczcg5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-sgrs2v/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc0009b8bc0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-sgrs2v" in resourcegroup="rg-np-version-upgrade-sgrs2v-l46fqcfczcg5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-sgrs2v-l46fqcfczcg5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-sgrs2v/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:178]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc001069ba8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-171841, targetMinor=5.1.0-0.nightly-multi-2026-09-30-105558) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-30-105558"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:178]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc001069ba8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-171841, targetMinor=5.1.0-0.nightly-multi-2026-09-30-105558) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-30-105558"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:198]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
    <*errors.joinError | 0xc001619020>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:198]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
    <*errors.joinError | 0xc001619020>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc001472300>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4zkcv5" resourcegroup="rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4zkcv5/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc001472300>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4zkcv5" resourcegroup="rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4zkcv5/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf"
Unexpected error:
    <*fmt.wrapError | 0xc000ca4c80>: 
    failed to create NodePool np-fzw8hf: failed starting nodepool creation "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf" in resourcegroup="rg-nightly-4-20-tfqsvtwn4ns7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-tfqsvtwn4ns7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-fzw8hf/nodePools/np-fzw8hf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf"
Unexpected error:
    <*fmt.wrapError | 0xc000ca4c80>: 
    failed to create NodePool np-fzw8hf: failed starting nodepool creation "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf" in resourcegroup="rg-nightly-4-20-tfqsvtwn4ns7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-tfqsvtwn4ns7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-fzw8hf/nodePools/np-fzw8hf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0009c85c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-wbpxbvspwkdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0",
      "name": "44b3f004-e16f-436f-bdc7-4652956b17b0",
      "status": "Failed",
      "startTime": "2026-09-30T02:12:50.954793928Z",
      "endTime": "2026-09-30T02:31:50.125456331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mhjdk-jdk87: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-mhjdk-jdk87: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0009c85c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-wbpxbvspwkdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0",
      "name": "44b3f004-e16f-436f-bdc7-4652956b17b0",
      "status": "Failed",
      "startTime": "2026-09-30T02:12:50.954793928Z",
      "endTime": "2026-09-30T02:31:50.125456331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mhjdk-jdk87: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-mhjdk-jdk87: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc001415050>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc001415050>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-srbb2hbvsczt/cluster-nightly-5-1-vxb9dm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c1af40>: 
    failed to create HCP cluster cluster-nightly-5-1-vxb9dm: failed waiting for cluster="cluster-nightly-5-1-vxb9dm" in resourcegroup="rg-nightly-5-1-srbb2hbvsczt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "name": "fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "status": "Failed",
      "startTime": "2026-09-30T23:26:50.189510673Z",
      "endTime": "2026-09-30T23:55:57.188151115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-105558 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-srbb2hbvsczt/cluster-nightly-5-1-vxb9dm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c1af40>: 
    failed to create HCP cluster cluster-nightly-5-1-vxb9dm: failed waiting for cluster="cluster-nightly-5-1-vxb9dm" in resourcegroup="rg-nightly-5-1-srbb2hbvsczt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "name": "fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "status": "Failed",
      "startTime": "2026-09-30T23:26:50.189510673Z",
      "endTime": "2026-09-30T23:55:57.188151115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-105558 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-prg87r" for cluster "cluster-nightly-4-20-prg87r"
Unexpected error:
    <*fmt.wrapError | 0xc0014907c0>: 
    failed to create NodePool np-prg87r: failed starting nodepool creation "np-prg87r" for cluster "cluster-nightly-4-20-prg87r" in resourcegroup="rg-nightly-4-20-kjrkpmc2vcnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kjrkpmc2vcnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-prg87r/nodePools/np-prg87r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-prg87r" for cluster "cluster-nightly-4-20-prg87r"
Unexpected error:
    <*fmt.wrapError | 0xc0014907c0>: 
    failed to create NodePool np-prg87r: failed starting nodepool creation "np-prg87r" for cluster "cluster-nightly-4-20-prg87r" in resourcegroup="rg-nightly-4-20-kjrkpmc2vcnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kjrkpmc2vcnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-prg87r/nodePools/np-prg87r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc00154c140>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-49bq5x" resourcegroup="rg-np-dg-minor-49bq5x-7f6vt9z24c7x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-49bq5x-7f6vt9z24c7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-49bq5x/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc00154c140>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-49bq5x" resourcegroup="rg-np-dg-minor-49bq5x-7f6vt9z24c7x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-49bq5x-7f6vt9z24c7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-49bq5x/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000b92100>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-qglcmw" in resourcegroup="rg-np-version-upgrade-qglcmw-2xxcc62c56zb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-qglcmw-2xxcc62c56zb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-qglcmw/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000b92100>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-qglcmw" in resourcegroup="rg-np-version-upgrade-qglcmw-2xxcc62c56zb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-qglcmw-2xxcc62c56zb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-qglcmw/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000d74240>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-qlzv6l" in resourcegroup="rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qlzv6l/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000d74240>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-qlzv6l" in resourcegroup="rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qlzv6l-d9kp5xhmf6nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qlzv6l/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-djrg27" for cluster "cluster-candidate-4-20-djrg27"
Unexpected error:
    <*fmt.wrapError | 0xc000119c40>: 
    failed to create NodePool np-djrg27: failed waiting for nodepool="np-djrg27" for cluster "cluster-candidate-4-20-djrg27" in resourcegroup="rg-candidate-4-20-hwlm5v4jsr9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "name": "94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "status": "Failed",
      "startTime": "2026-09-30T16:46:47.162303776Z",
      "endTime": "2026-09-30T17:05:53.284012473Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-djrg27" for cluster "cluster-candidate-4-20-djrg27"
Unexpected error:
    <*fmt.wrapError | 0xc000119c40>: 
    failed to create NodePool np-djrg27: failed waiting for nodepool="np-djrg27" for cluster "cluster-candidate-4-20-djrg27" in resourcegroup="rg-candidate-4-20-hwlm5v4jsr9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "name": "94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "status": "Failed",
      "startTime": "2026-09-30T16:46:47.162303776Z",
      "endTime": "2026-09-30T17:05:53.284012473Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc001703c60>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jmc9c7" in resourcegroup="rg-np-skip-minor-jmc9c7-9dtxqp29vbsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
      "name": "d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
      "status": "Failed",
      "startTime": "2026-09-30T17:21:41.15243756Z",
      "endTime": "2026-09-30T17:40:44.044506032Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc001703c60>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jmc9c7" in resourcegroup="rg-np-skip-minor-jmc9c7-9dtxqp29vbsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
      "name": "d4c4dc83-d9cb-477b-aaa0-0403e9664a6f",
      "status": "Failed",
      "startTime": "2026-09-30T17:21:41.15243756Z",
      "endTime": "2026-09-30T17:40:44.044506032Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jmc9c7-9dtxqp29vbsr-np-sm-jmc9c7--managed/providers/Microsoft.Compute/virtualMachines/c4f5p4i1c7o2n5j-nps-420-422-c7z6k-ms2xz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001116780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-vh7hggp8cw8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba",
      "name": "eed44f57-1a91-4587-8901-935588d355ba",
      "status": "Failed",
      "startTime": "2026-09-30T17:09:44.769920829Z",
      "endTime": "2026-09-30T17:28:44.009160044Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001116780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-vh7hggp8cw8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba",
      "name": "eed44f57-1a91-4587-8901-935588d355ba",
      "status": "Failed",
      "startTime": "2026-09-30T17:09:44.769920829Z",
      "endTime": "2026-09-30T17:28:44.009160044Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc0015c0ae0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-twtdf7gdskgb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "name": "ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "status": "Failed",
      "startTime": "2026-09-30T16:25:28.987424124Z",
      "endTime": "2026-09-30T16:44:30.182558055Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-njlb6-5n7mv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-njlb6-5n7mv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc0015c0ae0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-twtdf7gdskgb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "name": "ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "status": "Failed",
      "startTime": "2026-09-30T16:25:28.987424124Z",
      "endTime": "2026-09-30T16:44:30.182558055Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-njlb6-5n7mv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-njlb6-5n7mv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc0012c2500>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-xhfvnf" in resourcegroup="rg-np-noedge-xhfvnf-d6cdpdqmxzz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "name": "79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:42.49451817Z",
      "endTime": "2026-09-30T17:39:43.993225307Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc0012c2500>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-xhfvnf" in resourcegroup="rg-np-noedge-xhfvnf-d6cdpdqmxzz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "name": "79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:42.49451817Z",
      "endTime": "2026-09-30T17:39:43.993225307Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00148e9c0>: 
    resource group "rg-zstream-upgrade-5-0-l6m7gzggxh5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-mrptx7" in resourcegroup="rg-zstream-upgrade-5-0-l6m7gzggxh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "name": "d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "status": "Failed",
      "startTime": "2026-09-30T13:28:47.89996524Z",
      "endTime": "2026-09-30T13:52:47.080528249Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5bj2pp5rm42moqm553gsgpuq5afnl4 still exists (deletion dispatched at 2026-09-30T13:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00148e9c0>: 
    resource group "rg-zstream-upgrade-5-0-l6m7gzggxh5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-mrptx7" in resourcegroup="rg-zstream-upgrade-5-0-l6m7gzggxh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "name": "d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "status": "Failed",
      "startTime": "2026-09-30T13:28:47.89996524Z",
      "endTime": "2026-09-30T13:52:47.080528249Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5bj2pp5rm42moqm553gsgpuq5afnl4 still exists (deletion dispatched at 2026-09-30T13:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc0009ed920>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-rkbcqh" in resourcegroup="rg-np-skip-minor-rkbcqh-h7clvkbvfrzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
      "name": "0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
      "status": "Failed",
      "startTime": "2026-09-30T13:38:48.671242236Z",
      "endTime": "2026-09-30T13:57:57.057793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc0009ed920>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-rkbcqh" in resourcegroup="rg-np-skip-minor-rkbcqh-h7clvkbvfrzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
      "name": "0e29588c-4a53-4b7a-8ca9-501b18d6f6cb",
      "status": "Failed",
      "startTime": "2026-09-30T13:38:48.671242236Z",
      "endTime": "2026-09-30T13:57:57.057793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-rkbcqh-h7clvkbvfrzv-np-sm-rkbcqh--managed/providers/Microsoft.Compute/virtualMachines/i3m1n5y2h7p6d9x-nps-420-422-mhgzh-94492 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000c62580>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4ldsbj98fhm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693",
      "name": "d5b038c9-d01f-4cce-8da4-821ff1fad693",
      "status": "Failed",
      "startTime": "2026-09-30T12:41:58.878961581Z",
      "endTime": "2026-09-30T13:01:07.023295945Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000c62580>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4ldsbj98fhm7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5b038c9-d01f-4cce-8da4-821ff1fad693",
      "name": "d5b038c9-d01f-4cce-8da4-821ff1fad693",
      "status": "Failed",
      "startTime": "2026-09-30T12:41:58.878961581Z",
      "endTime": "2026-09-30T13:01:07.023295945Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4ldsbj98fhm7--managed/providers/Microsoft.Compute/virtualMachines/p9k9r7n9n9q9i7t-np-1-kxfwv-jwz82 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cd79a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ldfdqgv9q9rv" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cd79a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ldfdqgv9q9rv" to finish creating: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a8a4b0>: 
    resource group "rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-b6jrz8" in resourcegroup="rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "name": "8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "status": "Failed",
      "startTime": "2026-09-30T08:07:19.758629115Z",
      "endTime": "2026-09-30T08:31:25.87901072Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56tcneq332c0ht4fv2g22jot5kdluc still exists (deletion dispatched at 2026-09-30T08:07:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a8a4b0>: 
    resource group "rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-b6jrz8" in resourcegroup="rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "name": "8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "status": "Failed",
      "startTime": "2026-09-30T08:07:19.758629115Z",
      "endTime": "2026-09-30T08:31:25.87901072Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56tcneq332c0ht4fv2g22jot5kdluc still exists (deletion dispatched at 2026-09-30T08:07:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0013950a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6stx4mwz4j4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "name": "626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "status": "Failed",
      "startTime": "2026-09-30T08:05:30.277326796Z",
      "endTime": "2026-09-30T08:24:30.136180053Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0013950a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6stx4mwz4j4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "name": "626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "status": "Failed",
      "startTime": "2026-09-30T08:05:30.277326796Z",
      "endTime": "2026-09-30T08:24:30.136180053Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lssckk" for cluster "cluster-candidate-4-22-lssckk"
Unexpected error:
    <*fmt.wrapError | 0xc000d0d380>: 
    failed to create NodePool np-lssckk: failed waiting for nodepool="np-lssckk" for cluster "cluster-candidate-4-22-lssckk" in resourcegroup="rg-candidate-4-22-bqwkx6ssnq7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8",
      "name": "b6149906-5b1c-417e-9e9c-3b2c8a352da8",
      "status": "Failed",
      "startTime": "2026-09-30T02:38:56.070433953Z",
      "endTime": "2026-09-30T02:57:58.4595049Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lssckk" for cluster "cluster-candidate-4-22-lssckk"
Unexpected error:
    <*fmt.wrapError | 0xc000d0d380>: 
    failed to create NodePool np-lssckk: failed waiting for nodepool="np-lssckk" for cluster "cluster-candidate-4-22-lssckk" in resourcegroup="rg-candidate-4-22-bqwkx6ssnq7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b6149906-5b1c-417e-9e9c-3b2c8a352da8",
      "name": "b6149906-5b1c-417e-9e9c-3b2c8a352da8",
      "status": "Failed",
      "startTime": "2026-09-30T02:38:56.070433953Z",
      "endTime": "2026-09-30T02:57:58.4595049Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-bqwkx6ssnq7c-candidate-lssckk--managed/providers/Microsoft.Compute/virtualMachines/e6g5l0a6t3i7x6g-np-lssckk-rm7bq-b6c64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc001035640>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-4kv8jbqzgm6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27",
      "name": "0011df08-d47e-405c-b953-36672f501c27",
      "status": "Failed",
      "startTime": "2026-09-30T03:17:50.983140083Z",
      "endTime": "2026-09-30T03:36:56.457139595Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc001035640>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-4kv8jbqzgm6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0011df08-d47e-405c-b953-36672f501c27",
      "name": "0011df08-d47e-405c-b953-36672f501c27",
      "status": "Failed",
      "startTime": "2026-09-30T03:17:50.983140083Z",
      "endTime": "2026-09-30T03:36:56.457139595Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-4kv8jbqzgm6h--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-lm6c8-swzpr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0008827a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tb59h9fbmsnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985",
      "name": "69cf1894-8af7-4f4d-985a-7d39b5ffb985",
      "status": "Failed",
      "startTime": "2026-09-30T02:15:06.409498278Z",
      "endTime": "2026-09-30T02:34:10.118150935Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0008827a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-tb59h9fbmsnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69cf1894-8af7-4f4d-985a-7d39b5ffb985",
      "name": "69cf1894-8af7-4f4d-985a-7d39b5ffb985",
      "status": "Failed",
      "startTime": "2026-09-30T02:15:06.409498278Z",
      "endTime": "2026-09-30T02:34:10.118150935Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-tb59h9fbmsnp--managed/providers/Microsoft.Compute/virtualMachines/o2v5c2m5n1d2o2y-np-1-dbbj9-65tmd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016d54b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016d54b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c25d80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jtns6kb9n72s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521",
      "name": "13d22eca-d7af-43b2-9ab7-e630c81c0521",
      "status": "Failed",
      "startTime": "2026-09-30T02:16:05.748001285Z",
      "endTime": "2026-09-30T02:35:10.119119459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c25d80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jtns6kb9n72s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13d22eca-d7af-43b2-9ab7-e630c81c0521",
      "name": "13d22eca-d7af-43b2-9ab7-e630c81c0521",
      "status": "Failed",
      "startTime": "2026-09-30T02:16:05.748001285Z",
      "endTime": "2026-09-30T02:35:10.119119459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jtns6kb9n72s--managed/providers/Microsoft.Compute/virtualMachines/q5y5f0n3v7s7d3l-np-1-c6z9g-tg2mg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred