alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:50Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0504704-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.78:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0504704/providers/microsoft.monitor/accounts/services-j0504704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5fdbbd5c5b-l6bt7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0504704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:11:31Z
  Ended:       2026-10-08T23:17:33Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0504704-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:04:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3153024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.30:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3153024/providers/microsoft.monitor/accounts/hcps-j3153024", namespace="ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-tbmch", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:04:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3153024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.30:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3153024/providers/microsoft.monitor/accounts/hcps-j3153024", namespace="ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-tbmch", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat52el7ut34bj4g2ggf0vutik7dd90-j3v2u2r2o5q0x4m/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-7fcmrmvxxv7n/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-7fcmrmvxxv7n/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4990592-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.37:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4990592/providers/microsoft.monitor/accounts/services-j4990592", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c59bdf7c8-spr9r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4990592-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:31:34Z
  Ended:       2026-10-09T00:04:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4990592-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j4990592-svc).

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2c4b80612378>: 
    resource group "ingress-cert-le-68vr9hbvh2st": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-68vr9hbvh2st" to finish deleting: GET https://rp.j3544576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050",
      "name": "ca06aaf0-429b-4724-9342-8885fc527050",
      "status": "Failed",
      "startTime": "2026-10-08T23:18:31.808957419Z",
      "endTime": "2026-10-08T23:42:33.737902461Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tat5kosr5birnmo003enu4ag9tvkblp still exists (deletion dispatched at 2026-10-08T23:18:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2c4b80612378>: 
    resource group "ingress-cert-le-68vr9hbvh2st": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-68vr9hbvh2st" to finish deleting: GET https://rp.j3544576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca06aaf0-429b-4724-9342-8885fc527050",
      "name": "ca06aaf0-429b-4724-9342-8885fc527050",
      "status": "Failed",
      "startTime": "2026-10-08T23:18:31.808957419Z",
      "endTime": "2026-10-08T23:42:33.737902461Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tat5kosr5birnmo003enu4ag9tvkblp still exists (deletion dispatched at 2026-10-08T23:18:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 6 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:00:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.220:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vt6m4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat5kosr5birnmo003enu4ag9tvkblp-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:16:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-sst2s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:17:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-sst2s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T23:16:25Z
  Ended:       2026-10-08T23:32:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3544576-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:01:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="cloud-network-config-controller-79b774bd6b-9pkrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6b4e41e1-2282-44a1-ab27-d2b7bffdbb7d"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller-79b774bd6b-9pkrz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:01:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="csi-snapshot-controller-7fcf9f696b-5dd5v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ade41fab-9ae3-4c1c-af6d-d25dc46782e7"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/csi-snapshot-controller-7fcf9f696b-5dd5v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="csi-snapshot-controller-7fcf9f696b-5dd5v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ade41fab-9ae3-4c1c-af6d-d25dc46782e7"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/csi-snapshot-controller-7fcf9f696b-5dd5v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T00:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3544576-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3544576/providers/microsoft.monitor/accounts/hcps-j3544576", namespace="ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster", pod="cloud-network-config-controller-79b774bd6b-9pkrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6b4e41e1-2282-44a1-ab27-d2b7bffdbb7d"
  Description: Pod ocm-arohcpci01-2tat6eoal9j7k661kv4fcopvbul662l3-basic-cluster/cloud-network-config-controller-79b774bd6b-9pkrz (controller) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x14550d98c9a0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-p98fbdr6lfb6": PUT https://rp.j6887424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p98fbdr6lfb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x14550d98c9a0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-p98fbdr6lfb6": PUT https://rp.j6887424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p98fbdr6lfb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1044cd538270>: 
    resource group "rg-hs-pre--l7mzsgckn5bq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-m4xdbw" in resourcegroup="rg-hs-pre--l7mzsgckn5bq" to finish deleting: GET https://rp.j8864256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33",
      "name": "81f1c7e5-bd06-48b6-8e00-126748584d33",
      "status": "Failed",
      "startTime": "2026-10-08T22:52:33.320294122Z",
      "endTime": "2026-10-08T23:16:35.834245337Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tasnkqnpkt06eq3lmg9dsk6e9purnb2 still exists (deletion dispatched at 2026-10-08T22:52:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1044cd538270>: 
    resource group "rg-hs-pre--l7mzsgckn5bq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-m4xdbw" in resourcegroup="rg-hs-pre--l7mzsgckn5bq" to finish deleting: GET https://rp.j8864256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f1c7e5-bd06-48b6-8e00-126748584d33",
      "name": "81f1c7e5-bd06-48b6-8e00-126748584d33",
      "status": "Failed",
      "startTime": "2026-10-08T22:52:33.320294122Z",
      "endTime": "2026-10-08T23:16:35.834245337Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tasnkqnpkt06eq3lmg9dsk6e9purnb2 still exists (deletion dispatched at 2026-10-08T22:52:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-ttr5gf9j2z4p/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-ttr5gf9j2z4p/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp"
Unexpected error:
    <*fmt.wrapError | 0x36065ae1e700>: 
    failed to create NodePool np-2b5cwp: failed waiting for nodepool="np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp" in resourcegroup="rg-candidate-5-1-mdbffxtcx8wr" to finish creating: GET https://rp.j6247936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "name": "e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "status": "Failed",
      "startTime": "2026-10-08T19:25:31.299024517Z",
      "endTime": "2026-10-08T19:54:32.257674658Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp"
Unexpected error:
    <*fmt.wrapError | 0x36065ae1e700>: 
    failed to create NodePool np-2b5cwp: failed waiting for nodepool="np-2b5cwp" for cluster "cluster-candidate-5-1-2b5cwp" in resourcegroup="rg-candidate-5-1-mdbffxtcx8wr" to finish creating: GET https://rp.j6247936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "name": "e3f9fab1-4935-49c2-8e86-66ac4db0e7cb",
      "status": "Failed",
      "startTime": "2026-10-08T19:25:31.299024517Z",
      "endTime": "2026-10-08T19:54:32.257674658Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-mdbffxtcx8wr-candidate-2b5cwp--managed/providers/Microsoft.Compute/virtualMachines/e5w2m4n7q0t5w6r-np-2b5cwp-krd4h-vff46 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:58:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:59:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:07:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:07:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6247936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6247936/providers/microsoft.monitor/accounts/hcps-j6247936", namespace="ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-6l8pk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapqhcmll32asjjo23us2mqtb5k3kur-b2t0m2c3a0t5y9q/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 11 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-2g4tl", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-2g4tl has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-l5mn9", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-l5mn9 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vfshb", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vfshb has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-qhgdx", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-qhgdx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-wm888", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-wm888 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-t9gmx", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-t9gmx has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-h7lkm", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-h7lkm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-w5s5h", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-w5s5h has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vwbm7", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vwbm7 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-n644b", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-n644b has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:06:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8981120-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-gh9b9", region="westus3", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-gh9b9 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:07:32Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j5978496-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.233:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5978496/providers/microsoft.monitor/accounts/services-j5978496", namespace="klusterlet-2tapn8dlbj9p40f8ecef335ah1n44qgl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-snrs6", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tapn8dlbj9p40f8ecef335ah1n44qgl on cluster ci01-j5978496-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:22:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:23:48Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8343424-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.84:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8343424/providers/microsoft.monitor/accounts/services-j8343424", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68b89b8957-rg9qs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8343424-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T19:18:13Z
  Ended:       2026-10-08T19:24:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8343424-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x1d9f3c92fc0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gcnbkpsxtjrv" to finish creating: GET https://rp.j5951872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "name": "511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "status": "Failed",
      "startTime": "2026-10-08T20:56:10.667754986Z",
      "endTime": "2026-10-08T21:25:14.560841698Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x1d9f3c92fc0>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-gcnbkpsxtjrv" to finish creating: GET https://rp.j5951872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "name": "511caec2-c197-4b9c-9a43-b8c00295c7d1",
      "status": "Failed",
      "startTime": "2026-10-08T20:56:10.667754986Z",
      "endTime": "2026-10-08T21:25:14.560841698Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-gn6m6 to exist\nMachine e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-gcnbkpsxtjrv--managed/providers/Microsoft.Compute/virtualMachines/e1k5o2y0i4g4i1e-cordon-np-dpcqc-kfh5v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T21:07:52Z
  Ended:       2026-10-08T21:13:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5951872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tar6kfvsbfkgd4sfo2s5ak9up9mghnd-ephemeral-disk", pod="openshift-apiserver-f7bf46fd6-jsjg9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tar6kfvsbfkgd4sfo2s5ak9up9mghnd-ephemeral-disk/openshift-apiserver-f7bf46fd6-jsjg9 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T19:16:33Z
  Ended:       2026-10-08T19:22:34Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6052992-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:58:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6052992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6052992/providers/microsoft.monitor/accounts/hcps-j6052992", namespace="ocm-arohcpci01-2tapkvkaamqqutsmtseph2ro2qdkc8vv-d3m0g1e7k9t9s9o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vc7b6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapkvkaamqqutsmtseph2ro2qdkc8vv-d3m0g1e7k9t9s9o/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:01:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6052992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6052992/providers/microsoft.monitor/accounts/hcps-j6052992", namespace="ocm-arohcpci01-2tapkvkaamqqutsmtseph2ro2qdkc8vv-d3m0g1e7k9t9s9o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vc7b6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapkvkaamqqutsmtseph2ro2qdkc8vv-d3m0g1e7k9t9s9o/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T20:18:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6052992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6052992/providers/microsoft.monitor/accounts/hcps-j6052992", namespace="ocm-arohcpci01-2tapkvkaamqqutsmtseph2ro2qdkc8vv-d3m0g1e7k9t9s9o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vc7b6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tapkvkaamqqutsmtseph2ro2qdkc8vv-d3m0g1e7k9t9s9o/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4tll47xvzcmb/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4tll47xvzcmb/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:44:43Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7043968-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.30:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7043968/providers/microsoft.monitor/accounts/services-j7043968", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6fdbbdc5f8-bb886", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7043968-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T19:44:43Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7043968-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.30:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7043968/providers/microsoft.monitor/accounts/services-j7043968", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-6fdbbdc5f8-bb886", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7043968-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
  }
}
--------------------------------------------------------------------------------

github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0x17a8dd744af0?, {0x7f91c02f24c0?, 0x17a8dd261560?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1075 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0x17a8dcfdf200, {0x7f91c02f24c0, 0x17a8dd261560}, {0xf9f89cb, 0x15}, {0x17a8dc60e129, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:508 +0x2a8
github.com/Azure/ARO-HCP/test/e2e.init.func34.2({0x7f91c02f24c0, 0x17a8dd261560})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:68 +0x139
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x101a3e60?, 0x17a8dd261560?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 134
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xdfb

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked: failed to get subscription ID: ClientSecretCredential: unable to resolve an endpoint: server response error:
 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0x17a8dd744af0?, {0x7f91c02f24c0?, 0x17a8dd261560?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:1075 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0x17a8dcfdf200, {0x7f91c02f24c0, 0x17a8dd261560}, {0xf9f89cb, 0x15}, {0x17a8dc60e129, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:508 +0x2a8
github.com/Azure/ARO-HCP/test/e2e.init.func34.2({0x7f91c02f24c0, 0x17a8dd261560})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:68 +0x139
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x101a3e60?, 0x17a8dd261560?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 134
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xdfb

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked: failed to get subscription ID: ClientSecretCredential: unable to resolve an endpoint: server response error:
 context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:476]: Test Panicked

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T18:47:49Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9147264-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.232:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9147264/providers/microsoft.monitor/accounts/services-j9147264", namespace="klusterlet-2taok2013r1ut7fvo0ulo5kc534ntq7s", pod="arohcp-monitor-kube-state-metrics-7988dc648c-fljb2", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2taok2013r1ut7fvo0ulo5kc534ntq7s on cluster ci01-j9147264-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T18:18:15Z
  Ended:       2026-10-08T18:25:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9147264-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2taok4lfsu899t43cjah6nqtn60rbr6d-r8d4l6w3w4r4h6t", pod="openshift-apiserver-59b5875fdc-r9dd5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2taok4lfsu899t43cjah6nqtn60rbr6d-r8d4l6w3w4r4h6t/openshift-apiserver-59b5875fdc-r9dd5 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T18:03:54Z
  Ended:       2026-10-08T18:09:56Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9147264-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xed922aab58>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0xed9195f1e0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0xed91fd22c0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xed922aab58>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0xed9195f1e0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0xed91fd22c0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1c054b45c180>: 
    resource group "rg-candidate-4-21-l7wm67p4w7mm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-pjnsh8" in resourcegroup="rg-candidate-4-21-l7wm67p4w7mm" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac",
      "name": "de22b374-9b97-455f-b396-0c997a7b79ac",
      "status": "Failed",
      "startTime": "2026-10-08T17:28:41.684717507Z",
      "endTime": "2026-10-08T17:52:48.530098677Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1c054b45c180>: 
    resource group "rg-candidate-4-21-l7wm67p4w7mm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-pjnsh8" in resourcegroup="rg-candidate-4-21-l7wm67p4w7mm" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de22b374-9b97-455f-b396-0c997a7b79ac",
      "name": "de22b374-9b97-455f-b396-0c997a7b79ac",
      "status": "Failed",
      "startTime": "2026-10-08T17:28:41.684717507Z",
      "endTime": "2026-10-08T17:52:48.530098677Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126c34d780d8>: 
    resource group "rg-np-version-upgrade-d2kz66-n2vxth4prxxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-d2kz66" in resourcegroup="rg-np-version-upgrade-d2kz66-n2vxth4prxxw" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "name": "53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "status": "Failed",
      "startTime": "2026-10-08T17:58:56.699498743Z",
      "endTime": "2026-10-08T18:23:02.391989177Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tao6fb6en2d65csg2b2sr0ei58fvhru still exists (deletion dispatched at 2026-10-08T17:58:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126c34d780d8>: 
    resource group "rg-np-version-upgrade-d2kz66-n2vxth4prxxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-d2kz66" in resourcegroup="rg-np-version-upgrade-d2kz66-n2vxth4prxxw" to finish deleting: GET https://rp.j8427264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "name": "53f3ea8e-927e-4f49-ab09-36a6f2f31c6f",
      "status": "Failed",
      "startTime": "2026-10-08T17:58:56.699498743Z",
      "endTime": "2026-10-08T18:23:02.391989177Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tao6fb6en2d65csg2b2sr0ei58fvhru still exists (deletion dispatched at 2026-10-08T17:58:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

time=2026-10-08T16:19:05.365Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T16:30:17.342Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T16:19:05.375275155Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T16:19:08.114395254Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T16:19:08.462375095Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-08T16:30:17.335389178Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"6F2FEEE9F3FAC221\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j5048320-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:19:14.8133695Z\",\"duration\":\"PT9M5.9123598S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:30:17.335532727Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"98034B352C012EE7\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j5048320-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT4.7536001S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:30:17.33558093Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DE129F3B0C485FAF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j5048320-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT1M38.1712414S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:30:17.335599002Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B325BEA3E7A07E7A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j5048320-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT1M56.0129258S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:30:17.335620623Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"D7A0CEADFD719B73\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j5048320-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:28:15.8471301Z\",\"duration\":\"PT3.9567105S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:30:17.335657835Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7F61B179B809D260\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1\",\"resourceName\":\"ci01-j5048320-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:19:14.8133695Z\",\"duration\":\"PT9M0.9591654S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:30:17.335709239Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5048320-mgmt-1\",\"clusterName\":\"ci01-j5048320-mgmt-1\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584101305363180932\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5048320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5048320-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 72 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"11m11.960448525s\"}\n exit status 1"

time=2026-10-08T16:14:31.581Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T16:22:39.946Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T16:14:31.590606721Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T16:14:33.50353792Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T16:14:33.891298669Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-08T16:22:39.944584337Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"241855BF9A0D266F\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j3418880-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT1M38.1681956S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:22:39.944653829Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"5B81E75ADA721336\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j3418880-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT4.9210349S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:22:39.94468287Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"0A9A6CB955E8A5E9\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j3418880-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:14:38.0622167Z\",\"duration\":\"PT7M50.9531763S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:22:39.944693871Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"127F3C3755EDC782\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j3418880-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT4.2491544S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:22:39.944712361Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1AF780C01A639DF0\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j3418880-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-08T16:20:32.6231206Z\",\"duration\":\"PT4.2335315S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-08T16:22:39.944730542Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2DBAB0983C574AF5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1\",\"resourceName\":\"ci01-j3418880-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-08T16:14:38.0622167Z\",\"duration\":\"PT5M54.4662985S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-08T16:22:39.944752483Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3418880-mgmt-1\",\"clusterName\":\"ci01-j3418880-mgmt-1\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584101308111130568\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      },\\n      {\\n        \\\"code\\\": \\\"BadRequest\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3418880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3418880-mgmt-1/agentPools/userswft2\\\",\\n        \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"8m8.354161621s\"}\n exit status 1"

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T17:46:24Z
  Ended:       2026-10-08T19:30:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8250240-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.87:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8250240/providers/microsoft.monitor/accounts/services-j8250240", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5d9b69f6ff-z8bjk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8250240-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Resolved
  Started:     2026-10-08T17:46:24Z
  Ended:       2026-10-08T19:30:23Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8250240-svc", cluster_type="svc-cluster", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.87:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8250240/providers/microsoft.monitor/accounts/services-j8250240", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5d9b69f6ff-z8bjk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8250240-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb"
Unexpected error:
    <*fmt.wrapError | 0x1e49d4d5a340>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb" in resourcegroup="rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj" to finish creating: GET https://rp.j0335104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646",
      "name": "b0463f50-61f7-4515-9352-022d3d794646",
      "status": "Failed",
      "startTime": "2026-10-08T16:41:51.052187772Z",
      "endTime": "2026-10-08T17:10:57.967451326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb"
Unexpected error:
    <*fmt.wrapError | 0x1e49d4d5a340>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-n4rtkb" in resourcegroup="rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj" to finish creating: GET https://rp.j0335104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0463f50-61f7-4515-9352-022d3d794646",
      "name": "b0463f50-61f7-4515-9352-022d3d794646",
      "status": "Failed",
      "startTime": "2026-10-08T16:41:51.052187772Z",
      "endTime": "2026-10-08T17:10:57.967451326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-zvs4xjzt6ggj-cp-ystream-den-6158ac1e/providers/Microsoft.Compute/virtualMachines/r4e3r4w5k9d2k3l-np-deny-pods-z474z-5l4nk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng"
Unexpected error:
    <*fmt.wrapError | 0x1dd12c4c3a80>: 
    failed to create NodePool np-ft6nng: failed waiting for nodepool="np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng" in resourcegroup="rg-candidate-4-22-j25t2jm5j66l" to finish creating: GET https://rp.j9390848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng"
Unexpected error:
    <*fmt.wrapError | 0x1dd12c4c3a80>: 
    failed to create NodePool np-ft6nng: failed waiting for nodepool="np-ft6nng" for cluster "cluster-candidate-4-22-ft6nng" in resourcegroup="rg-candidate-4-22-j25t2jm5j66l" to finish creating: GET https://rp.j9390848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1bff9fd2-6231-420f-ace0-63d110f1901e"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2e1efc3180c0>: 
    resource group "admin-api-serialconsole-bootdiag-2vphmzvmmzbh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-2vphmzvmmzbh" to finish deleting: GET https://rp.j4279424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "name": "7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:07.770334262Z",
      "endTime": "2026-10-08T16:41:10.588488129Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tan1stnb9qf5nfuks8bh5mvhk8g2eul still exists (deletion dispatched at 2026-10-08T16:17:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2e1efc3180c0>: 
    resource group "admin-api-serialconsole-bootdiag-2vphmzvmmzbh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-2vphmzvmmzbh" to finish deleting: GET https://rp.j4279424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "name": "7a1b3c13-6b24-46ac-a6bd-8eadaaab84aa",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:07.770334262Z",
      "endTime": "2026-10-08T16:41:10.588488129Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tan1stnb9qf5nfuks8bh5mvhk8g2eul still exists (deletion dispatched at 2026-10-08T16:17:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3adaf74760a8>: 
    resource group "fips-enabled-khqp8vkktpzt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-khqp8vkktpzt" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "name": "407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "status": "Failed",
      "startTime": "2026-10-08T16:13:27.229044406Z",
      "endTime": "2026-10-08T16:37:32.626253468Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamus8q8drvpor1j4v73qnjn2pbcpag still exists (deletion dispatched at 2026-10-08T16:13:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3adaf74760a8>: 
    resource group "fips-enabled-khqp8vkktpzt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-khqp8vkktpzt" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "name": "407d6cdf-c346-4727-b3ca-9d3137b70cc4",
      "status": "Failed",
      "startTime": "2026-10-08T16:13:27.229044406Z",
      "endTime": "2026-10-08T16:37:32.626253468Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamus8q8drvpor1j4v73qnjn2pbcpag still exists (deletion dispatched at 2026-10-08T16:13:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-l22klm" for cluster "cluster-candidate-4-21-l22klm"
Unexpected error:
    <*fmt.wrapError | 0x320d00821e20>: 
    failed to create NodePool np-l22klm: failed waiting for nodepool="np-l22klm" for cluster "cluster-candidate-4-21-l22klm" in resourcegroup="rg-candidate-4-21-hqlkfl7j7kk5" to finish creating: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7ada65d-8a80-4fa7-a9ff-9aa084567ce4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7ada65d-8a80-4fa7-a9ff-9aa084567ce4",
      "name": "e7ada65d-8a80-4fa7-a9ff-9aa084567ce4",
      "status": "Failed",
      "startTime": "2026-10-08T16:12:49.167614937Z",
      "endTime": "2026-10-08T16:41:52.222456636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn to exist\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn to exist\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-l22klm" for cluster "cluster-candidate-4-21-l22klm"
Unexpected error:
    <*fmt.wrapError | 0x320d00821e20>: 
    failed to create NodePool np-l22klm: failed waiting for nodepool="np-l22klm" for cluster "cluster-candidate-4-21-l22klm" in resourcegroup="rg-candidate-4-21-hqlkfl7j7kk5" to finish creating: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7ada65d-8a80-4fa7-a9ff-9aa084567ce4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7ada65d-8a80-4fa7-a9ff-9aa084567ce4",
      "name": "e7ada65d-8a80-4fa7-a9ff-9aa084567ce4",
      "status": "Failed",
      "startTime": "2026-10-08T16:12:49.167614937Z",
      "endTime": "2026-10-08T16:41:52.222456636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn to exist\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-sgrrn to exist\nMachine c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hqlkfl7j7kk5-candidate-l22klm--managed/providers/Microsoft.Compute/virtualMachines/c4a9q0q2m9f5u5s-np-l22klm-wxdn5-txk4j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1aa3a56380a8>: 
    resource group "private-kas-mmbkx8tpnrlv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-mmbkx8tpnrlv" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494",
      "name": "bb91148d-7554-4a55-b1dd-4854fa37b494",
      "status": "Failed",
      "startTime": "2026-10-08T16:22:19.235799645Z",
      "endTime": "2026-10-08T16:46:22.547532555Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamuoo5fqaudkk7q5op01b86l5utmqh still exists (deletion dispatched at 2026-10-08T16:22:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1aa3a56380a8>: 
    resource group "private-kas-mmbkx8tpnrlv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-mmbkx8tpnrlv" to finish deleting: GET https://rp.j2446464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bb91148d-7554-4a55-b1dd-4854fa37b494",
      "name": "bb91148d-7554-4a55-b1dd-4854fa37b494",
      "status": "Failed",
      "startTime": "2026-10-08T16:22:19.235799645Z",
      "endTime": "2026-10-08T16:46:22.547532555Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tamuoo5fqaudkk7q5op01b86l5utmqh still exists (deletion dispatched at 2026-10-08T16:22:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-l8tn4qdvlqv4/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-l8tn4qdvlqv4/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x44478a7c150>: 
    resource group "arm64-vm-cluster-ldnw4gs8dwrw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ldnw4gs8dwrw" to finish deleting: GET https://rp.j7078784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1",
      "name": "fcf937e7-61c6-4778-aabd-893fd37924c1",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:57.9406496Z",
      "endTime": "2026-10-08T16:41:57.233664583Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x44478a7c150>: 
    resource group "arm64-vm-cluster-ldnw4gs8dwrw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ldnw4gs8dwrw" to finish deleting: GET https://rp.j7078784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcf937e7-61c6-4778-aabd-893fd37924c1",
      "name": "fcf937e7-61c6-4778-aabd-893fd37924c1",
      "status": "Failed",
      "startTime": "2026-10-08T16:17:57.9406496Z",
      "endTime": "2026-10-08T16:41:57.233664583Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

time=2026-10-08T14:39:49.369Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-08T14:39:51.193Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-08T14:39:53.050Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4570d5430c346685aef785faa645617cca8a23038712be321dc348c5ac428645 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1446656-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4570d5430c346685aef785faa645617cca8a23038712be321dc348c5ac428645
time=2026-10-08T15:09:49.053Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T15:04:42Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j1446656-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T15:05:40Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j1446656-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x255cb079f740>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dgrj4944q7jq" to finish creating: GET https://rp.j5460992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "name": "5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "status": "Failed",
      "startTime": "2026-10-08T15:30:05.316469426Z",
      "endTime": "2026-10-08T15:59:10.872777291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x255cb079f740>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-dgrj4944q7jq" to finish creating: GET https://rp.j5460992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "name": "5a819b58-a3c9-46ef-aa03-af80037d2c41",
      "status": "Failed",
      "startTime": "2026-10-08T15:30:05.316469426Z",
      "endTime": "2026-10-08T15:59:10.872777291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-dgrj4944q7jq--managed/providers/Microsoft.Compute/virtualMachines/l4c0q3f2g2d3k3l-np-delete-main-gqjhl-tns94 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T16:08:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5460992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.17:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5460992/providers/microsoft.monitor/accounts/hcps-j5460992", namespace="ocm-arohcpci01-2tam3ufd6i9i7hbiu868p9lhfmmiku8i-v4d4z2a2t7q6o5b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jxjxn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tam3ufd6i9i7hbiu868p9lhfmmiku8i-v4d4z2a2t7q6o5b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T16:08:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5460992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.17:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5460992/providers/microsoft.monitor/accounts/hcps-j5460992", namespace="ocm-arohcpci01-2tam3ufd6i9i7hbiu868p9lhfmmiku8i-v4d4z2a2t7q6o5b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jxjxn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tam3ufd6i9i7hbiu868p9lhfmmiku8i-v4d4z2a2t7q6o5b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

time=2026-10-08T14:32:31.059Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-08T15:02:30.832Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-08T14:32:31.068088679Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-08T14:32:32.982339672Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-08T14:32:33.61885433Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4452224-mgmt-1\",\"clusterName\":\"ci01-j4452224-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T14:57:45Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4452224-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:312]: Timed out after 2700.006s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:310 with:
console should return a successful response or redirect, got 503
Expected
    <int>: 503
to be <
    <int>: 400
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:312]: Timed out after 2700.006s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:310 with:
console should return a successful response or redirect, got 503
Expected
    <int>: 503
to be <
    <int>: 400

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1c27e024540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-6zjcv2t4zcfg" to finish creating: GET https://rp.j0298368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "name": "7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "status": "Failed",
      "startTime": "2026-10-08T14:08:53.200082246Z",
      "endTime": "2026-10-08T14:38:00.180447244Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1c27e024540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-6zjcv2t4zcfg" to finish creating: GET https://rp.j0298368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "name": "7fdab8b1-d3ef-4d1c-89fb-9da45a5eaf01",
      "status": "Failed",
      "startTime": "2026-10-08T14:08:53.200082246Z",
      "endTime": "2026-10-08T14:38:00.180447244Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0x119e8d352a00>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);
    let _endTime = datetime(2026-10-08T14:29:33.7915314Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7dxbsk584tbw'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T14:29:42.5063608Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666", ActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", SubActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityType:"GW.Http.CallContext", ParentActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityStack:"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x119e8dac0f30>{
            errs: [
                <*fmt.wrapError | 0x119e8d3529e0>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x119e8d3529c0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x119e8d36b280>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x119e8d36b1a0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x119e8b843000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0x119e8d352a00>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);
    let _endTime = datetime(2026-10-08T14:29:33.7915314Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-7dxbsk584tbw'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-08T14:29:42.5063608Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666", ActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", SubActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityType:"GW.Http.CallContext", ParentActivityId:"21a4f26b-f20b-4845-825d-4d44325cd58e", ActivityStack:"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x119e8dac0f30>{
            errs: [
                <*fmt.wrapError | 0x119e8d3529e0>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x119e8d3529c0>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x119e8d36b280>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x119e8d36b1a0>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-07T14:29:33.7915312Z);\nlet _endTime = datetime(2026-10-08T14:29:33.7915314Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-7dxbsk584tbw'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-08T14:29:42.5063608Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666\", ActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", SubActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"21a4f26b-f20b-4845-825d-4d44325cd58e\", ActivityStack:\"(Activity stack: CRID=KGC.execute;2d601710-400e-4f38-8a97-e4a8f4407666 ARID=21a4f26b-f20b-4845-825d-4d44325cd58e > GW.Http.CallContext/21a4f26b-f20b-4845-825d-4d44325cd58e)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x119e8b843000>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T15:27:53Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7737472-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.93:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7737472/providers/microsoft.monitor/accounts/services-j7737472", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-67bdf9f749-q2zbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7737472-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T15:27:53Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7737472-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.93:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7737472/providers/microsoft.monitor/accounts/services-j7737472", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-67bdf9f749-q2zbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7737472-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x1657ab71ce40>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-rkxd84qqqgg8": PUT https://rp.j9463680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rkxd84qqqgg8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x1657ab71ce40>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-rkxd84qqqgg8": PUT https://rp.j9463680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-rkxd84qqqgg8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j7950976' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j7950976' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7950976/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j7950976' could not be found."
  }
}
--------------------------------------------------------------------------------

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-54jpd8" for cluster "cluster-candidate-5-1-54jpd8"
Unexpected error:
    <*fmt.wrapError | 0x18bafeade620>: 
    failed to create NodePool np-54jpd8: failed waiting for nodepool="np-54jpd8" for cluster "cluster-candidate-5-1-54jpd8" in resourcegroup="rg-candidate-5-1-ch7fc46qsw9p" to finish creating: GET https://rp.j9653888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b250e4db-d610-4d88-ac6c-923c943777b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b250e4db-d610-4d88-ac6c-923c943777b1",
      "name": "b250e4db-d610-4d88-ac6c-923c943777b1",
      "status": "Failed",
      "startTime": "2026-10-08T13:07:33.724991852Z",
      "endTime": "2026-10-08T13:36:35.125261835Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll to exist\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll to exist\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-54jpd8" for cluster "cluster-candidate-5-1-54jpd8"
Unexpected error:
    <*fmt.wrapError | 0x18bafeade620>: 
    failed to create NodePool np-54jpd8: failed waiting for nodepool="np-54jpd8" for cluster "cluster-candidate-5-1-54jpd8" in resourcegroup="rg-candidate-5-1-ch7fc46qsw9p" to finish creating: GET https://rp.j9653888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b250e4db-d610-4d88-ac6c-923c943777b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b250e4db-d610-4d88-ac6c-923c943777b1",
      "name": "b250e4db-d610-4d88-ac6c-923c943777b1",
      "status": "Failed",
      "startTime": "2026-10-08T13:07:33.724991852Z",
      "endTime": "2026-10-08T13:36:35.125261835Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll to exist\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-cshll to exist\nMachine a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-ch7fc46qsw9p-candidate-54jpd8--managed/providers/Microsoft.Compute/virtualMachines/a9w9k0t2v4a5r8d-np-54jpd8-ctzwh-nqwtj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x3dc0df3e260>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x3dc0df3e260>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:52922->172.30.0.10:53: read: connection refused",
    }

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:48:14Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9653888-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.155:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9653888/providers/microsoft.monitor/accounts/services-j9653888", namespace="klusterlet-2tak8n7sjj3mgsij1id8amijusdtem6a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-87ql4", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tak8n7sjj3mgsij1id8amijusdtem6a on cluster ci01-j9653888-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T13:09:26Z
  Ended:       2026-10-08T13:15:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9653888-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T13:21:47Z
  Ended:       2026-10-08T13:27:52Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3932160-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1ae233ff2078>: 
    resource group "v20260901preview-hqg6wzs5tgtz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-hqg6wzs5tgtz" to finish deleting: GET https://rp.j2977792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951",
      "name": "b7f165f5-05d4-46b6-9b64-0134316e8951",
      "status": "Failed",
      "startTime": "2026-10-08T12:28:19.710418085Z",
      "endTime": "2026-10-08T12:52:26.28519628Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tajia53q41l8grpaq7k8eu3qvn81ojn still exists (deletion dispatched at 2026-10-08T12:28:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1ae233ff2078>: 
    resource group "v20260901preview-hqg6wzs5tgtz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-hqg6wzs5tgtz" to finish deleting: GET https://rp.j2977792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f165f5-05d4-46b6-9b64-0134316e8951",
      "name": "b7f165f5-05d4-46b6-9b64-0134316e8951",
      "status": "Failed",
      "startTime": "2026-10-08T12:28:19.710418085Z",
      "endTime": "2026-10-08T12:52:26.28519628Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tajia53q41l8grpaq7k8eu3qvn81ojn still exists (deletion dispatched at 2026-10-08T12:28:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:12:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2977792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2977792/providers/microsoft.monitor/accounts/hcps-j2977792", namespace="ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gngm8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:12:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2977792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2977792/providers/microsoft.monitor/accounts/hcps-j2977792", namespace="ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gngm8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:12:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2977792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2977792/providers/microsoft.monitor/accounts/hcps-j2977792", namespace="ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gngm8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:12:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2977792-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2977792/providers/microsoft.monitor/accounts/hcps-j2977792", namespace="ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gngm8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tajia53q41l8grpaq7k8eu3qvn81ojn-v20260901/cluster-api has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T12:21:55Z
  Ended:       2026-10-08T12:27:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2977792-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:21:27Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j2954368-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.126:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2954368/providers/microsoft.monitor/accounts/services-j2954368", namespace="klusterlet-2tajh66uh8t9dscan4deki4hln9el07b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-kbbmz", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tajh66uh8t9dscan4deki4hln9el07b on cluster ci01-j2954368-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T12:19:34Z
  Ended:       2026-10-08T12:30:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2954368-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:160]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0x2d2a1467ee00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-t4pdz4zt795h" to finish creating: GET https://rp.j3688704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5864c8a-d5bc-47ca-a843-6cbfb4efb9b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5864c8a-d5bc-47ca-a843-6cbfb4efb9b0",
      "name": "b5864c8a-d5bc-47ca-a843-6cbfb4efb9b0",
      "status": "Failed",
      "startTime": "2026-10-08T12:56:17.97671191Z",
      "endTime": "2026-10-08T13:25:25.246131122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-t4pdz4zt795h--managed/providers/Microsoft.Compute/virtualMachines/q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-t4pdz4zt795h--managed/providers/Microsoft.Compute/virtualMachines/q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:160]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0x2d2a1467ee00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-t4pdz4zt795h" to finish creating: GET https://rp.j3688704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5864c8a-d5bc-47ca-a843-6cbfb4efb9b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5864c8a-d5bc-47ca-a843-6cbfb4efb9b0",
      "name": "b5864c8a-d5bc-47ca-a843-6cbfb4efb9b0",
      "status": "Failed",
      "startTime": "2026-10-08T12:56:17.97671191Z",
      "endTime": "2026-10-08T13:25:25.246131122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-t4pdz4zt795h--managed/providers/Microsoft.Compute/virtualMachines/q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-t4pdz4zt795h--managed/providers/Microsoft.Compute/virtualMachines/q0i4e6e7j3u1g3m-np-1-mkz4r-gn4c2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:31:01Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j3688704-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.89:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3688704/providers/microsoft.monitor/accounts/services-j3688704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-77896677c6-bt8h2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j3688704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T13:31:01Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j3688704-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.89:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3688704/providers/microsoft.monitor/accounts/services-j3688704", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-77896677c6-bt8h2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j3688704-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T12:26:40Z
  Ended:       2026-10-08T12:42:41Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3688704-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T12:01:31Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7008768-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.41:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7008768/providers/microsoft.monitor/accounts/services-j7008768", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-58dfcbcb48-7bd6w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7008768-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-08T12:01:31Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7008768-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.41:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7008768/providers/microsoft.monitor/accounts/services-j7008768", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-58dfcbcb48-7bd6w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7008768-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T10:49:03Z
  Ended:       2026-10-08T10:55:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7008768-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x1749b81ac3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-jlkcb2vkhpvr" to finish creating: GET https://rp.j8876032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdaa0297-3302-494a-9a7f-438dfbd8479c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdaa0297-3302-494a-9a7f-438dfbd8479c",
      "name": "cdaa0297-3302-494a-9a7f-438dfbd8479c",
      "status": "Failed",
      "startTime": "2026-10-08T10:40:01.811608698Z",
      "endTime": "2026-10-08T11:09:08.599114626Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-9xb9s-glzcx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-jlkcb2vkhpvr--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-9xb9s-glzcx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-9xb9s-glzcx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-jlkcb2vkhpvr--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-9xb9s-glzcx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x1749b81ac3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-jlkcb2vkhpvr" to finish creating: GET https://rp.j8876032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdaa0297-3302-494a-9a7f-438dfbd8479c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdaa0297-3302-494a-9a7f-438dfbd8479c",
      "name": "cdaa0297-3302-494a-9a7f-438dfbd8479c",
      "status": "Failed",
      "startTime": "2026-10-08T10:40:01.811608698Z",
      "endTime": "2026-10-08T11:09:08.599114626Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-9xb9s-glzcx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-jlkcb2vkhpvr--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-9xb9s-glzcx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-9xb9s-glzcx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-jlkcb2vkhpvr--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-9xb9s-glzcx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T10:35:41Z
  Ended:       2026-10-08T10:41:38Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1895552-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T10:15:16Z
  Ended:       2026-10-08T10:21:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8870784-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x39ccc1989460>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-p2kczkmtdgbx" to finish creating: GET https://rp.j5271040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f832996-ec93-465f-b6ee-4ab328dac405
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f832996-ec93-465f-b6ee-4ab328dac405",
      "name": "1f832996-ec93-465f-b6ee-4ab328dac405",
      "status": "Failed",
      "startTime": "2026-10-08T10:21:11.725925281Z",
      "endTime": "2026-10-08T10:50:14.636681892Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-zbjzt-gngqj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-p2kczkmtdgbx--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zbjzt-gngqj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-zbjzt-gngqj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-p2kczkmtdgbx--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zbjzt-gngqj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x39ccc1989460>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-p2kczkmtdgbx" to finish creating: GET https://rp.j5271040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f832996-ec93-465f-b6ee-4ab328dac405
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f832996-ec93-465f-b6ee-4ab328dac405",
      "name": "1f832996-ec93-465f-b6ee-4ab328dac405",
      "status": "Failed",
      "startTime": "2026-10-08T10:21:11.725925281Z",
      "endTime": "2026-10-08T10:50:14.636681892Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-zbjzt-gngqj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-p2kczkmtdgbx--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zbjzt-gngqj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-zbjzt-gngqj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-p2kczkmtdgbx--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zbjzt-gngqj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jc5rgf" for cluster "cluster-candidate-4-20-jc5rgf"
Unexpected error:
    <*fmt.wrapError | 0x49efa775a40>: 
    failed to create NodePool np-jc5rgf: failed waiting for nodepool="np-jc5rgf" for cluster "cluster-candidate-4-20-jc5rgf" in resourcegroup="rg-candidate-4-20-cbxcrrxtz89m" to finish creating: GET https://rp.j5271040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5aac86d-773c-4950-b645-5bbd5ad5a70a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5aac86d-773c-4950-b645-5bbd5ad5a70a",
      "name": "c5aac86d-773c-4950-b645-5bbd5ad5a70a",
      "status": "Failed",
      "startTime": "2026-10-08T10:23:03.365232367Z",
      "endTime": "2026-10-08T10:52:04.576957009Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-cbxcrrxtz89m-candidate-jc5rgf--managed/providers/Microsoft.Compute/virtualMachines/y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-cbxcrrxtz89m-candidate-jc5rgf--managed/providers/Microsoft.Compute/virtualMachines/y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jc5rgf" for cluster "cluster-candidate-4-20-jc5rgf"
Unexpected error:
    <*fmt.wrapError | 0x49efa775a40>: 
    failed to create NodePool np-jc5rgf: failed waiting for nodepool="np-jc5rgf" for cluster "cluster-candidate-4-20-jc5rgf" in resourcegroup="rg-candidate-4-20-cbxcrrxtz89m" to finish creating: GET https://rp.j5271040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5aac86d-773c-4950-b645-5bbd5ad5a70a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5aac86d-773c-4950-b645-5bbd5ad5a70a",
      "name": "c5aac86d-773c-4950-b645-5bbd5ad5a70a",
      "status": "Failed",
      "startTime": "2026-10-08T10:23:03.365232367Z",
      "endTime": "2026-10-08T10:52:04.576957009Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-cbxcrrxtz89m-candidate-jc5rgf--managed/providers/Microsoft.Compute/virtualMachines/y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-cbxcrrxtz89m-candidate-jc5rgf--managed/providers/Microsoft.Compute/virtualMachines/y0h4n6k5k2x2i5v-np-jc5rgf-nhkc9-rxczq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3a17b89ea108>: 
    resource group "rg-candidate-5-0-m7pk2bqx25bp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-8cn6hj" in resourcegroup="rg-candidate-5-0-m7pk2bqx25bp" to finish deleting: GET https://rp.j8241280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8e4a1e-473e-4006-96fb-1eb1247636a3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8e4a1e-473e-4006-96fb-1eb1247636a3",
      "name": "ae8e4a1e-473e-4006-96fb-1eb1247636a3",
      "status": "Failed",
      "startTime": "2026-10-08T10:05:41.110442631Z",
      "endTime": "2026-10-08T10:29:50.911284161Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2taheibf9i4u8stp8ms5pnmlpeb5go3l still exists (deletion dispatched at 2026-10-08T10:05:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3a17b89ea108>: 
    resource group "rg-candidate-5-0-m7pk2bqx25bp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-8cn6hj" in resourcegroup="rg-candidate-5-0-m7pk2bqx25bp" to finish deleting: GET https://rp.j8241280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8e4a1e-473e-4006-96fb-1eb1247636a3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae8e4a1e-473e-4006-96fb-1eb1247636a3",
      "name": "ae8e4a1e-473e-4006-96fb-1eb1247636a3",
      "status": "Failed",
      "startTime": "2026-10-08T10:05:41.110442631Z",
      "endTime": "2026-10-08T10:29:50.911284161Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2taheibf9i4u8stp8ms5pnmlpeb5go3l still exists (deletion dispatched at 2026-10-08T10:05:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xd2898b7c180>: 
    resource group "private-kas-f55dwxz4lh8f": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-f55dwxz4lh8f" to finish deleting: GET https://rp.j8241280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "name": "f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "status": "Failed",
      "startTime": "2026-10-08T10:06:03.074876755Z",
      "endTime": "2026-10-08T10:30:10.535297041Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tahe6f6jvv1r5au11pgmqub5ah0te26 still exists (deletion dispatched at 2026-10-08T10:06:03Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xd2898b7c180>: 
    resource group "private-kas-f55dwxz4lh8f": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-f55dwxz4lh8f" to finish deleting: GET https://rp.j8241280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "name": "f521bfd4-8e62-4bc6-af54-f7aeca38d8ed",
      "status": "Failed",
      "startTime": "2026-10-08T10:06:03.074876755Z",
      "endTime": "2026-10-08T10:30:10.535297041Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tahe6f6jvv1r5au11pgmqub5ah0te26 still exists (deletion dispatched at 2026-10-08T10:06:03Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T09:56:24Z
  Ended:       2026-10-08T10:02:27Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8241280-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-85g664x7dr6x/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-85g664x7dr6x/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0x32468342f3e0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-wtnllf" in resourcegroup="rg-np-noedge-wtnllf-hjnhf58nl2hn" to finish creating: GET https://rp.j4302720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4471bd89-63c3-4d52-ab14-11262ab0d01e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4471bd89-63c3-4d52-ab14-11262ab0d01e",
      "name": "4471bd89-63c3-4d52-ab14-11262ab0d01e",
      "status": "Failed",
      "startTime": "2026-10-08T10:10:30.503055567Z",
      "endTime": "2026-10-08T10:39:39.262233637Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-wtnllf-hjnhf58nl2hn-np-ne-wtnllf--managed/providers/Microsoft.Compute/virtualMachines/p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-wtnllf-hjnhf58nl2hn-np-ne-wtnllf--managed/providers/Microsoft.Compute/virtualMachines/p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0x32468342f3e0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-wtnllf" in resourcegroup="rg-np-noedge-wtnllf-hjnhf58nl2hn" to finish creating: GET https://rp.j4302720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4471bd89-63c3-4d52-ab14-11262ab0d01e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4471bd89-63c3-4d52-ab14-11262ab0d01e",
      "name": "4471bd89-63c3-4d52-ab14-11262ab0d01e",
      "status": "Failed",
      "startTime": "2026-10-08T10:10:30.503055567Z",
      "endTime": "2026-10-08T10:39:39.262233637Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-wtnllf-hjnhf58nl2hn-np-ne-wtnllf--managed/providers/Microsoft.Compute/virtualMachines/p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-wtnllf-hjnhf58nl2hn-np-ne-wtnllf--managed/providers/Microsoft.Compute/virtualMachines/p4p6y7m3b1v5s6t-npnoedge-4-20-7bmz2-b9qj6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-bzbvdd52lfmz/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-bzbvdd52lfmz/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x1409e4db1080>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1409e4324dc0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1409e4f4c060>{
                    msg: "DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1409e4f4c040>{
                        msg: "DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0x1409e4db1080>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1409e4324dc0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1409e4f4c060>{
                    msg: "DNS for route host agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1409e4f4c040>{
                        msg: "DNS for agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-76bcc.apps.aro.cilium-cluster.brv0.j5059200.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T09:47:35Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1758080-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.67:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1758080/providers/microsoft.monitor/accounts/services-j1758080", namespace="klusterlet-2tage7jjs69ome454gjasrbup2nm9mab", pod="arohcp-monitor-kube-state-metrics-7988dc648c-whw96", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tage7jjs69ome454gjasrbup2nm9mab on cluster ci01-j1758080-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x18aecfdf0aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dq2w5s6mnvnb" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "name": "5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "status": "Failed",
      "startTime": "2026-10-08T08:17:53.86568203Z",
      "endTime": "2026-10-08T08:18:02.050800903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x18aecfdf0aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dq2w5s6mnvnb" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "name": "5591fbb4-44d5-4678-9638-e11ff2fb9020",
      "status": "Failed",
      "startTime": "2026-10-08T08:17:53.86568203Z",
      "endTime": "2026-10-08T08:18:02.050800903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xda1a20062a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-lbppg5gfcqw8" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f1d411e-4147-426c-8a46-dece07469237
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f1d411e-4147-426c-8a46-dece07469237",
      "name": "5f1d411e-4147-426c-8a46-dece07469237",
      "status": "Failed",
      "startTime": "2026-10-08T08:09:00.258803996Z",
      "endTime": "2026-10-08T08:38:02.86327598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-zz94m-k596d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-lbppg5gfcqw8--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zz94m-k596d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-zz94m-k596d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-lbppg5gfcqw8--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zz94m-k596d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xda1a20062a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-lbppg5gfcqw8" to finish creating: GET https://rp.j9632000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f1d411e-4147-426c-8a46-dece07469237
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f1d411e-4147-426c-8a46-dece07469237",
      "name": "5f1d411e-4147-426c-8a46-dece07469237",
      "status": "Failed",
      "startTime": "2026-10-08T08:09:00.258803996Z",
      "endTime": "2026-10-08T08:38:02.86327598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-zz94m-k596d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-lbppg5gfcqw8--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zz94m-k596d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-zz94m-k596d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-lbppg5gfcqw8--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zz94m-k596d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-08T08:59:55Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9632000-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.149:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9632000/providers/microsoft.monitor/accounts/services-j9632000", namespace="klusterlet-2tafv3kk3g55204rl52vhttrl8vlq1g0", pod="arohcp-monitor-kube-state-metrics-7988dc648c-hq5xt", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tafv3kk3g55204rl52vhttrl8vlq1g0 on cluster ci01-j9632000-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0xef3ce514120>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.007s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0xef3ce514100>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0xef3ce5140e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x7b70a422fc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x7b70a4e87a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x7b70a6fbd20>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x7b70a422fc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x7b70a4e87a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x7b70a6fbd20>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-zj96vcc7brng/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-zj96vcc7brng/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xa118982a360>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dx54wv2l48hl" to finish creating: GET https://rp.j4542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "name": "54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "status": "Failed",
      "startTime": "2026-10-08T05:35:12.262324182Z",
      "endTime": "2026-10-08T06:04:17.39866058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xa118982a360>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-dx54wv2l48hl" to finish creating: GET https://rp.j4542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "name": "54ed370f-4e3b-4fe8-92dd-a7f286d6f761",
      "status": "Failed",
      "startTime": "2026-10-08T05:35:12.262324182Z",
      "endTime": "2026-10-08T06:04:17.39866058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T06:09:36Z
  Ended:       2026-10-08T06:23:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-g29zr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T05:54:34Z
  Ended:       2026-10-08T06:18:34Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-g29zr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-08T05:54:34Z
  Ended:       2026-10-08T06:19:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-g29zr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T06:00:31Z
  Ended:       2026-10-08T06:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9e83f444-711c-486c-b496-8152e0711540"
  Description: Pod ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-08T06:00:31Z
  Ended:       2026-10-08T06:22:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4542976-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4542976/providers/microsoft.monitor/accounts/hcps-j4542976", namespace="ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9e83f444-711c-486c-b496-8152e0711540"
  Description: Pod ocm-arohcpci01-2tadr4bm6brdefckebujuar6b3a54pbe-agg-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T05:48:26Z
  Ended:       2026-10-08T06:05:24Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4542976-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x32ec947f60c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-mkxhsl7j2nvb" to finish creating: GET https://rp.j2143872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2ab5edd-253e-4c79-8950-79decb992f4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2ab5edd-253e-4c79-8950-79decb992f4b",
      "name": "b2ab5edd-253e-4c79-8950-79decb992f4b",
      "status": "Failed",
      "startTime": "2026-10-08T04:56:23.95109392Z",
      "endTime": "2026-10-08T05:25:32.665732941Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-zg8r4-bhhzf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-mkxhsl7j2nvb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zg8r4-bhhzf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-zg8r4-bhhzf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-mkxhsl7j2nvb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zg8r4-bhhzf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x32ec947f60c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-mkxhsl7j2nvb" to finish creating: GET https://rp.j2143872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2ab5edd-253e-4c79-8950-79decb992f4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2ab5edd-253e-4c79-8950-79decb992f4b",
      "name": "b2ab5edd-253e-4c79-8950-79decb992f4b",
      "status": "Failed",
      "startTime": "2026-10-08T04:56:23.95109392Z",
      "endTime": "2026-10-08T05:25:32.665732941Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-zg8r4-bhhzf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-mkxhsl7j2nvb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zg8r4-bhhzf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-zg8r4-bhhzf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-mkxhsl7j2nvb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-zg8r4-bhhzf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T05:01:06Z
  Ended:       2026-10-08T05:07:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2143872-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T04:04:43Z
  Ended:       2026-10-08T04:10:44Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5015552-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 57 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-gvdlf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-gvdlf has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-pj78n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-pj78n has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-jnjtd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-jnjtd has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-rplxb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-rplxb has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-htmxm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-htmxm has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-rv6cc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-rv6cc has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="cloud-network-config-controller-74769fccfb-k8kwp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/cloud-network-config-controller-74769fccfb-k8kwp has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-hgjs7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-hgjs7 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-gs5jc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-gs5jc has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-8chb5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-8chb5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-psbgs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-psbgs has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-4tbx4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-4tbx4 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-gjpp8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-gjpp8 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-j44kk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-j44kk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-mvbj5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-mvbj5 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-ss6rg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-ss6rg has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-snf5p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-snf5p has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-mdgsc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-mdgsc has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-cwkvx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-cwkvx has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-bnmrb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-bnmrb has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-cv4g6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-cv4g6 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-7n652", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-7n652 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-4hlqb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-4hlqb has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-w2spp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-w2spp has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-87cjv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-87cjv has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-ht9c9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-ht9c9 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-jwnrm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-jwnrm has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-8mmlh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-8mmlh has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-7pzw8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-7pzw8 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-672v2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-672v2 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-9c8996664-xqlcz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-9c8996664-xqlcz has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-f8gbx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-f8gbx has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-l29xs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-l29xs has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-mcs4j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-mcs4j has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-m8sbd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-m8sbd has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-fg868", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-fg868 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-d57l7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-d57l7 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-6qjzf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-6qjzf has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-ssj25", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-ssj25 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:36:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-2nvzx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-2nvzx has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-4rxvd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-4rxvd has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-l6dz6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-l6dz6 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-6qjld", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-6qjld has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-m9rtt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-m9rtt has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-2njx7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-2njx7 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-2t8lh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-2t8lh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-08T04:06:15Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-rhccp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-rhccp has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-g64ns", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-g64ns has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-5c9wj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-5c9wj has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-kdtjv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-kdtjv has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-5fld5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-5fld5 has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-skdmq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-skdmq has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-lfqpn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-lfqpn has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-85f67c86d5-kq58q", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-85f67c86d5-kq58q has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-pjnm6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-pjnm6 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-file-csi-driver-controller-c955dbc8f-hztch", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-file-csi-driver-controller-c955dbc8f-hztch has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-08T04:06:16Z
  Ended:       2026-10-08T04:31:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5015552-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a", pod="azure-disk-csi-driver-controller-d4df64db4-zgk6p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tac93jjbc7p84jirpisg4dmju2f1ram-w3k7u8o1f6o9c5a/azure-disk-csi-driver-controller-d4df64db4-zgk6p has been in a non-ready state for longer than 5 minutes.

time=2026-10-08T03:31:09.066Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n"
time=2026-10-08T03:31:09.688Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpci01 kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:c5d026c4a8335c2d098ce04776e25babc3e7773e70817b5b709f388bb52488d2] metricLabelsAllowlist:[nodes=[node.kubernetes.io/instance-type,kubernetes.azure.com/agentpool]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[system]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:9c285c83d939df7d0bdf27287562bb6fcf8fd6ab639d899abba1f14a0c2d5647] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:5fc5d09d0245cc1e7c6432ca9b0ebcc652c047b550d0289ea45725c14343585d]] thanosImage:map[registry:arohcpsvcdev.azurecr.io repository:thanos/thanos sha:b567818fe608067eb0f1d7c2c4fe361e7ad83c8a256234c97685f1d0bf670cc8 tag:v0.41.0] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[serviceAccount:map[managedIdentity:0b00177b-a54e-4160-b95f-d8ba55b89f42]] prometheusSpec:map[externalLabels:map[cluster:ci01-j0275840-svc environment:ci01 region:westus3] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:7ef9c1d80e4e8e1eec1a207166af22930c8f11200f25001d1a3285b30b4f13f9] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-ci01-j0275840-svc-ng1v.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-d00da46377ca403ab27fbe4e13d51efe/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 replicas:2 resources:map[limits:map[memory:NONE] requests:map[cpu:NONE memory:NONE]] shards:1 version: zoneCount:3] schedulingRole:system]"
time=2026-10-08T03:31:09.688Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-10-08T03:31:10.214Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus
time=2026-10-08T03:31:10.584Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false
time=2026-10-08T03:31:10.584Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-10-08T03:31:10.783Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus
time=2026-10-08T03:40:20.024Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-08T03:40:20.024Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"

time=2026-10-08T03:34:00.150Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-10-08T03:34:00.987Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-10-08T03:34:02.023Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=058182c57c37b78b09387fdf90aeaea50b1d526f842d6127dc4d5ebd76617220 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9563392-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F058182c57c37b78b09387fdf90aeaea50b1d526f842d6127dc4d5ebd76617220
time=2026-10-08T03:36:02.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.Resources/deployments/058182c57c37b78b09387fdf90aeaea50b1d526f842d6127dc4d5ebd76617220/operationStatuses/08584101764439358601\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-post\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"NotFound\\\",\\r\\n            \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/velero' was not found.\\\"\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"NotFound\\\",\\r\\n            \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mgmt-agent' was not found.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"NotFound\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"NotFound\\\",\\r\\n    \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9563392-mgmt-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/velero' was not found.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0x3e609f8540a8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-9c9j4)
    {
        errs: [
            <*fmt.wrapError | 0x3e609fd10180>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                err: <*fmt.wrapError | 0x3e609fd10160>{
                    msg: "failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                    err: <*fmt.wrapError | 0x3e609fd10020>{
                        msg: "failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                        err: <*errors.StatusError | 0x3e609eee2c80>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-9c9j4",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0x3e609f8540a8>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-9c9j4)
    {
        errs: [
            <*fmt.wrapError | 0x3e609fd10180>{
                msg: "VerifyFIPSEnabled failed: failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                err: <*fmt.wrapError | 0x3e609fd10160>{
                    msg: "failed to check FIPS mode on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4: failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                    err: <*fmt.wrapError | 0x3e609fd10020>{
                        msg: "failed to get pod logs for pod fips-check-9c9j4 in namespace e2e-fips-test-8h4th on node i8p2v2r0i5k3k0h-np-1-w6zbx-n7md4 after retries: an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                        err: <*errors.StatusError | 0x3e609eee2c80>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods fips-check-9c9j4)",
                                Reason: "InternalError",
                                Details: {
                                    Name: "fips-check-9c9j4",
                                    Group: "",
                                    Kind: "pods",
                                    UID: "",
                                    Causes: [
                                        {Type: ..., Message: ..., Field: ...},
                                    ],
                                    RetryAfterSeconds: 0,
                                },
                                Code: 500,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:444]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc4b0a06e918>: 
    failed to create NodePool agg-np-b: failed waiting for nodepool="agg-np-b" for cluster "agg-cluster" in resourcegroup="feature-aggregation-qnvb5555k8tb" to finish creating: GET https://rp.j8011648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "name": "7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "status": "Failed",
      "startTime": "2026-10-08T04:02:19.757714703Z",
      "endTime": "2026-10-08T04:31:24.163355482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:444]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc4b0a06e918>: 
    failed to create NodePool agg-np-b: failed waiting for nodepool="agg-np-b" for cluster "agg-cluster" in resourcegroup="feature-aggregation-qnvb5555k8tb" to finish creating: GET https://rp.j8011648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "name": "7b6e27a8-980f-45cb-ad53-cd8329dd5c4a",
      "status": "Failed",
      "startTime": "2026-10-08T04:02:19.757714703Z",
      "endTime": "2026-10-08T04:31:24.163355482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:435]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
    <*fmt.wrapError | 0x1d63e63ce540>: 
    VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1d63e63ce520>{
            msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1d63e63ce4e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e29e0f51bc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e29e2b846a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e29e176e4f0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0x1e29e0f51bc0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    {
        errs: [
            <*fmt.wrapError | 0x1e29e2b846a0>{
                msg: "VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil",
                err: <*errors.errorString | 0x1e29e176e4f0>{
                    s: "IngressController loadBalancer config is nil",
                },
            },
        ],
    }
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T03:22:03Z
  Ended:       2026-10-08T03:28:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2503552-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-08T03:04:48Z
  Ended:       2026-10-08T03:10:47Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9986432-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25f97d5e2360>: 
    resource group "tls-endpoint-cluster-z5gqw56sgjhd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-z5gqw56sgjhd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "name": "cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "status": "Failed",
      "startTime": "2026-10-08T21:40:36.777928134Z",
      "endTime": "2026-10-08T22:04:36.806096142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tarq2rh1nt9l8n2qiphckhb46c0kvk6 still exists (deletion dispatched at 2026-10-08T21:40:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x25f97d5e2360>: 
    resource group "tls-endpoint-cluster-z5gqw56sgjhd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-z5gqw56sgjhd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "name": "cc8151c4-e387-49ba-89c6-800d9fe39b7c",
      "status": "Failed",
      "startTime": "2026-10-08T21:40:36.777928134Z",
      "endTime": "2026-10-08T22:04:36.806096142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tarq2rh1nt9l8n2qiphckhb46c0kvk6 still exists (deletion dispatched at 2026-10-08T21:40:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:148]: failed to delete customer resource group "cx-rg-cluster-9528856dtms2"
Unexpected error:
    <*fmt.wrapErrors | 0x2df35190540>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded
    {
        msg: "failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2df35ff14c0>{
                s: "timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:148]: failed to delete customer resource group "cx-rg-cluster-9528856dtms2"
Unexpected error:
    <*fmt.wrapErrors | 0x2df35190540>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded
    {
        msg: "failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x2df35ff14c0>{
                s: "timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-9528856dtms2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x317d01c40258>: 
    resource group "v20260901preview-9cxtwb94v2tg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-9cxtwb94v2tg" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "name": "4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:49.176345278Z",
      "endTime": "2026-10-08T17:27:51.502199148Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmtqk63qrfodamveh24r6lp7p43kt still exists (deletion dispatched at 2026-10-08T17:03:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x317d01c40258>: 
    resource group "v20260901preview-9cxtwb94v2tg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-9cxtwb94v2tg" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "name": "4816c7c1-1d1e-40dc-9966-7fde6d93956a",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:49.176345278Z",
      "endTime": "2026-10-08T17:27:51.502199148Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmtqk63qrfodamveh24r6lp7p43kt still exists (deletion dispatched at 2026-10-08T17:03:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19f9b5e0a3f0>: 
    resource group "customer-rg-wkldzssbjxmj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-wkldzssbjxmj" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb82b4ed-34b1-4bae-b90f-ec9ebb76c7a8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb82b4ed-34b1-4bae-b90f-ec9ebb76c7a8",
      "name": "eb82b4ed-34b1-4bae-b90f-ec9ebb76c7a8",
      "status": "Failed",
      "startTime": "2026-10-08T17:08:36.973340805Z",
      "endTime": "2026-10-08T17:32:41.526425453Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tannaefijum1ouk23vffl0tmfuhorgq still exists (deletion dispatched at 2026-10-08T17:08:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19f9b5e0a3f0>: 
    resource group "customer-rg-wkldzssbjxmj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-wkldzssbjxmj" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb82b4ed-34b1-4bae-b90f-ec9ebb76c7a8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb82b4ed-34b1-4bae-b90f-ec9ebb76c7a8",
      "name": "eb82b4ed-34b1-4bae-b90f-ec9ebb76c7a8",
      "status": "Failed",
      "startTime": "2026-10-08T17:08:36.973340805Z",
      "endTime": "2026-10-08T17:32:41.526425453Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tannaefijum1ouk23vffl0tmfuhorgq still exists (deletion dispatched at 2026-10-08T17:08:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x63eda6241c8>: 
    resource group "rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "name": "c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:08.268369953Z",
      "endTime": "2026-10-08T17:27:11.496076999Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmuj8loas31r4p690qgf675h7c53q still exists (deletion dispatched at 2026-10-08T17:03:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x63eda6241c8>: 
    resource group "rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-sppg9t7zzmdm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "name": "c3f349e3-c307-4eb6-8abb-0bf949fe42f5",
      "status": "Failed",
      "startTime": "2026-10-08T17:03:08.268369953Z",
      "endTime": "2026-10-08T17:27:11.496076999Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tanmuj8loas31r4p690qgf675h7c53q still exists (deletion dispatched at 2026-10-08T17:03:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2700c25b4198>: 
    resource group "rg-cp-ystream-upgrade-5-1-7wd5gspw55gd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-1-wvvvhw" in resourcegroup="rg-cp-ystream-upgrade-5-1-7wd5gspw55gd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd1fd723-1818-4cf3-abd1-cd25fc17377b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd1fd723-1818-4cf3-abd1-cd25fc17377b",
      "name": "cd1fd723-1818-4cf3-abd1-cd25fc17377b",
      "status": "Failed",
      "startTime": "2026-10-08T01:41:07.332810869Z",
      "endTime": "2026-10-08T02:05:10.961421785Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2taa15pl8rhqv7goa573mk2hebqeei6j still exists (deletion dispatched at 2026-10-08T01:41:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2700c25b4198>: 
    resource group "rg-cp-ystream-upgrade-5-1-7wd5gspw55gd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-1-wvvvhw" in resourcegroup="rg-cp-ystream-upgrade-5-1-7wd5gspw55gd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd1fd723-1818-4cf3-abd1-cd25fc17377b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd1fd723-1818-4cf3-abd1-cd25fc17377b",
      "name": "cd1fd723-1818-4cf3-abd1-cd25fc17377b",
      "status": "Failed",
      "startTime": "2026-10-08T01:41:07.332810869Z",
      "endTime": "2026-10-08T02:05:10.961421785Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2taa15pl8rhqv7goa573mk2hebqeei6j still exists (deletion dispatched at 2026-10-08T01:41:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1a2d62cf4c18>: 
    resource group "node-limit-test-7ngnq4mvtzfd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-7ngnq4mvtzfd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7157c234-f4b7-4732-ad4a-89009867cbd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7157c234-f4b7-4732-ad4a-89009867cbd7",
      "name": "7157c234-f4b7-4732-ad4a-89009867cbd7",
      "status": "Failed",
      "startTime": "2026-10-08T01:35:15.90215739Z",
      "endTime": "2026-10-08T01:59:21.185260922Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2taa7pjbbs0aafhd6esu75djenuq7kvc still exists (deletion dispatched at 2026-10-08T01:35:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1a2d62cf4c18>: 
    resource group "node-limit-test-7ngnq4mvtzfd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-7ngnq4mvtzfd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7157c234-f4b7-4732-ad4a-89009867cbd7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7157c234-f4b7-4732-ad4a-89009867cbd7",
      "name": "7157c234-f4b7-4732-ad4a-89009867cbd7",
      "status": "Failed",
      "startTime": "2026-10-08T01:35:15.90215739Z",
      "endTime": "2026-10-08T01:59:21.185260922Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2taa7pjbbs0aafhd6esu75djenuq7kvc still exists (deletion dispatched at 2026-10-08T01:35:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1bc3bcaaef00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8cplvshb5svv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1bc3bcaaef00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8cplvshb5svv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x17466b892d50>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n8q8j5xgxdk4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x17466b892d50>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n8q8j5xgxdk4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2e62773595c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-m97nzsnh6v7k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2e62773595c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-m97nzsnh6v7k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3ea6fe4464b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rwz8zs2wwdcn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3ea6fe4464b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rwz8zs2wwdcn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2d03e6b69a40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-vv42cndqzc2q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2d03e6b69a40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-vv42cndqzc2q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2e7b26aac4c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fbkb2qz6f9rf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "name": "9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.271009954Z",
      "endTime": "2026-10-08T18:32:55.449378647Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2e7b26aac4c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fbkb2qz6f9rf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "name": "9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.271009954Z",
      "endTime": "2026-10-08T18:32:55.449378647Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cz595b" for cluster "cluster-candidate-4-22-cz595b"
Unexpected error:
    <*fmt.wrapError | 0x18c3c8280a0>: 
    failed to create NodePool np-cz595b: failed waiting for nodepool="np-cz595b" for cluster "cluster-candidate-4-22-cz595b" in resourcegroup="rg-candidate-4-22-vzqcrpffcth2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "name": "7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:25.022726186Z",
      "endTime": "2026-10-08T19:05:25.609844949Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cz595b" for cluster "cluster-candidate-4-22-cz595b"
Unexpected error:
    <*fmt.wrapError | 0x18c3c8280a0>: 
    failed to create NodePool np-cz595b: failed waiting for nodepool="np-cz595b" for cluster "cluster-candidate-4-22-cz595b" in resourcegroup="rg-candidate-4-22-vzqcrpffcth2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "name": "7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:25.022726186Z",
      "endTime": "2026-10-08T19:05:25.609844949Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4"
Unexpected error:
    <*fmt.wrapError | 0x1aded3148440>: 
    failed to create NodePool np-fn9nq4: failed waiting for nodepool="np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4" in resourcegroup="rg-candidate-4-21-ghsd6g8zdnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9",
      "name": "70a9d05d-3834-496b-ab23-90c8635291e9",
      "status": "Failed",
      "startTime": "2026-10-08T18:35:08.455675057Z",
      "endTime": "2026-10-08T19:04:15.635949535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4"
Unexpected error:
    <*fmt.wrapError | 0x1aded3148440>: 
    failed to create NodePool np-fn9nq4: failed waiting for nodepool="np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4" in resourcegroup="rg-candidate-4-21-ghsd6g8zdnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9",
      "name": "70a9d05d-3834-496b-ab23-90c8635291e9",
      "status": "Failed",
      "startTime": "2026-10-08T18:35:08.455675057Z",
      "endTime": "2026-10-08T19:04:15.635949535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2f509ecfe3c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bw4mbhgcm827" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "name": "d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "status": "Failed",
      "startTime": "2026-10-08T18:07:59.549473703Z",
      "endTime": "2026-10-08T18:37:05.482678031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2f509ecfe3c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bw4mbhgcm827" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "name": "d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "status": "Failed",
      "startTime": "2026-10-08T18:07:59.549473703Z",
      "endTime": "2026-10-08T18:37:05.482678031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n"
Unexpected error:
    <*fmt.wrapError | 0x35ae7eca0680>: 
    failed to create NodePool np-ntb88n: failed waiting for nodepool="np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n" in resourcegroup="rg-candidate-5-1-tvlhq8pw8r2j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122",
      "name": "00bc71ab-da38-456c-8cc5-ad615c84b122",
      "status": "Failed",
      "startTime": "2026-10-08T18:45:34.339197477Z",
      "endTime": "2026-10-08T19:14:37.105782951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n"
Unexpected error:
    <*fmt.wrapError | 0x35ae7eca0680>: 
    failed to create NodePool np-ntb88n: failed waiting for nodepool="np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n" in resourcegroup="rg-candidate-5-1-tvlhq8pw8r2j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122",
      "name": "00bc71ab-da38-456c-8cc5-ad615c84b122",
      "status": "Failed",
      "startTime": "2026-10-08T18:45:34.339197477Z",
      "endTime": "2026-10-08T19:14:37.105782951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3c50440e0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nkr4c2vp869x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e",
      "name": "5de3f7d0-158f-4a7e-a885-89292635394e",
      "status": "Failed",
      "startTime": "2026-10-08T18:08:57.323157264Z",
      "endTime": "2026-10-08T18:38:05.479695963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3c50440e0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nkr4c2vp869x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e",
      "name": "5de3f7d0-158f-4a7e-a885-89292635394e",
      "status": "Failed",
      "startTime": "2026-10-08T18:08:57.323157264Z",
      "endTime": "2026-10-08T18:38:05.479695963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb"
Unexpected error:
    <*fmt.wrapError | 0x2b65d70d0400>: 
    failed to create NodePool np-dlrjwb: failed waiting for nodepool="np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb" in resourcegroup="rg-candidate-4-23-sdtvdwbjbxfr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d",
      "name": "79fc773d-2a37-4106-8f63-f194f928388d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:39.156679392Z",
      "endTime": "2026-10-08T19:05:45.645587063Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb"
Unexpected error:
    <*fmt.wrapError | 0x2b65d70d0400>: 
    failed to create NodePool np-dlrjwb: failed waiting for nodepool="np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb" in resourcegroup="rg-candidate-4-23-sdtvdwbjbxfr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d",
      "name": "79fc773d-2a37-4106-8f63-f194f928388d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:39.156679392Z",
      "endTime": "2026-10-08T19:05:45.645587063Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c"
Unexpected error:
    <*fmt.wrapError | 0x553d3fef80>: 
    failed to create NodePool np-kn7h4c: failed waiting for nodepool="np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c" in resourcegroup="rg-candidate-5-0-5tp4pjp8hk9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "name": "b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "status": "Failed",
      "startTime": "2026-10-08T18:41:12.213982763Z",
      "endTime": "2026-10-08T19:10:15.712133624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c"
Unexpected error:
    <*fmt.wrapError | 0x553d3fef80>: 
    failed to create NodePool np-kn7h4c: failed waiting for nodepool="np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c" in resourcegroup="rg-candidate-5-0-5tp4pjp8hk9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "name": "b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "status": "Failed",
      "startTime": "2026-10-08T18:41:12.213982763Z",
      "endTime": "2026-10-08T19:10:15.712133624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm"
Unexpected error:
    <*fmt.wrapError | 0x3108b67364e0>: 
    failed to create NodePool np-8mqkrm: failed waiting for nodepool="np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm" in resourcegroup="rg-candidate-4-20-xmx872mb68nv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "name": "827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "status": "Failed",
      "startTime": "2026-10-08T18:32:58.020047255Z",
      "endTime": "2026-10-08T19:02:05.605513431Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm"
Unexpected error:
    <*fmt.wrapError | 0x3108b67364e0>: 
    failed to create NodePool np-8mqkrm: failed waiting for nodepool="np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm" in resourcegroup="rg-candidate-4-20-xmx872mb68nv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "name": "827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "status": "Failed",
      "startTime": "2026-10-08T18:32:58.020047255Z",
      "endTime": "2026-10-08T19:02:05.605513431Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x237e16e43180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-q28hc5z2kwsh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "name": "6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.204928396Z",
      "endTime": "2026-10-08T18:32:55.458196493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-j8vmw-6lgjc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-j8vmw-6lgjc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x237e16e43180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-q28hc5z2kwsh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "name": "6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.204928396Z",
      "endTime": "2026-10-08T18:32:55.458196493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-j8vmw-6lgjc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-j8vmw-6lgjc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk"
Unexpected error:
    <*fmt.wrapError | 0x39e342643720>: 
    failed to create NodePool np-qzm2gk: failed waiting for nodepool="np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk" in resourcegroup="rg-candidate-4-21-t5wksgdrrckp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "name": "afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "status": "Failed",
      "startTime": "2026-10-08T14:53:33.939084658Z",
      "endTime": "2026-10-08T15:22:35.324122041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk"
Unexpected error:
    <*fmt.wrapError | 0x39e342643720>: 
    failed to create NodePool np-qzm2gk: failed waiting for nodepool="np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk" in resourcegroup="rg-candidate-4-21-t5wksgdrrckp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "name": "afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "status": "Failed",
      "startTime": "2026-10-08T14:53:33.939084658Z",
      "endTime": "2026-10-08T15:22:35.324122041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x371874876980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-ftnkcdsss4wg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28",
      "name": "883d0d0d-c572-41f2-9f82-7d61a050db28",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:29.613460957Z",
      "endTime": "2026-10-08T14:56:34.939890509Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x371874876980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-ftnkcdsss4wg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28",
      "name": "883d0d0d-c572-41f2-9f82-7d61a050db28",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:29.613460957Z",
      "endTime": "2026-10-08T14:56:34.939890509Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x6335433a090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-tpwrtzjcnmjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "name": "3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:33.571990767Z",
      "endTime": "2026-10-08T14:56:35.04864797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-8swjn-292k4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-8swjn-292k4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x6335433a090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-tpwrtzjcnmjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "name": "3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:33.571990767Z",
      "endTime": "2026-10-08T14:56:35.04864797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-8swjn-292k4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-8swjn-292k4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4"
Unexpected error:
    <*fmt.wrapError | 0x3ac3b36042c0>: 
    failed to create NodePool np-ctg8q4: failed waiting for nodepool="np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4" in resourcegroup="rg-candidate-4-23-z87lsqfql56h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "name": "4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:55.181740971Z",
      "endTime": "2026-10-08T15:27:00.08772636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4"
Unexpected error:
    <*fmt.wrapError | 0x3ac3b36042c0>: 
    failed to create NodePool np-ctg8q4: failed waiting for nodepool="np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4" in resourcegroup="rg-candidate-4-23-z87lsqfql56h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "name": "4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:55.181740971Z",
      "endTime": "2026-10-08T15:27:00.08772636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a6cffe3f060>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jct5pzjh8mxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "name": "b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "status": "Failed",
      "startTime": "2026-10-08T15:48:06.632742492Z",
      "endTime": "2026-10-08T16:17:11.778582607Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a6cffe3f060>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jct5pzjh8mxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "name": "b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "status": "Failed",
      "startTime": "2026-10-08T15:48:06.632742492Z",
      "endTime": "2026-10-08T16:17:11.778582607Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk"
Unexpected error:
    <*fmt.wrapError | 0x36b6bf7fec00>: 
    failed to create NodePool np-24nxrk: failed waiting for nodepool="np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk" in resourcegroup="rg-candidate-5-1-74n6vxnsbzln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "name": "a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "status": "Failed",
      "startTime": "2026-10-08T14:58:11.889421655Z",
      "endTime": "2026-10-08T15:27:20.151153667Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk"
Unexpected error:
    <*fmt.wrapError | 0x36b6bf7fec00>: 
    failed to create NodePool np-24nxrk: failed waiting for nodepool="np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk" in resourcegroup="rg-candidate-5-1-74n6vxnsbzln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "name": "a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "status": "Failed",
      "startTime": "2026-10-08T14:58:11.889421655Z",
      "endTime": "2026-10-08T15:27:20.151153667Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8"
Unexpected error:
    <*fmt.wrapError | 0x2ad692bfa2c0>: 
    failed to create NodePool np-fgmnm8: failed waiting for nodepool="np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8" in resourcegroup="rg-candidate-4-22-t7czqbb9gdw6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "name": "df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "status": "Failed",
      "startTime": "2026-10-08T15:00:21.098278607Z",
      "endTime": "2026-10-08T15:29:30.220139004Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8"
Unexpected error:
    <*fmt.wrapError | 0x2ad692bfa2c0>: 
    failed to create NodePool np-fgmnm8: failed waiting for nodepool="np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8" in resourcegroup="rg-candidate-4-22-t7czqbb9gdw6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "name": "df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "status": "Failed",
      "startTime": "2026-10-08T15:00:21.098278607Z",
      "endTime": "2026-10-08T15:29:30.220139004Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x248495222160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-2s5lcfmgt854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921",
      "name": "93a34e6e-477c-4e65-9e08-527e17028921",
      "status": "Failed",
      "startTime": "2026-10-08T14:23:16.915326226Z",
      "endTime": "2026-10-08T14:52:17.249683042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-xxpw5-lhcfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-xxpw5-lhcfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x248495222160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-2s5lcfmgt854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921",
      "name": "93a34e6e-477c-4e65-9e08-527e17028921",
      "status": "Failed",
      "startTime": "2026-10-08T14:23:16.915326226Z",
      "endTime": "2026-10-08T14:52:17.249683042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-xxpw5-lhcfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-xxpw5-lhcfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm"
Unexpected error:
    <*fmt.wrapError | 0x1c37b4b9e200>: 
    failed to create NodePool np-fsgggm: failed waiting for nodepool="np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm" in resourcegroup="rg-candidate-4-20-rz65zsts87v9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068",
      "name": "240398ab-2ae4-4b7b-a355-04e9e2139068",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:34.48901399Z",
      "endTime": "2026-10-08T15:26:35.280704579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm"
Unexpected error:
    <*fmt.wrapError | 0x1c37b4b9e200>: 
    failed to create NodePool np-fsgggm: failed waiting for nodepool="np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm" in resourcegroup="rg-candidate-4-20-rz65zsts87v9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068",
      "name": "240398ab-2ae4-4b7b-a355-04e9e2139068",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:34.48901399Z",
      "endTime": "2026-10-08T15:26:35.280704579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2a3a9b00e6d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hxkght4ww89l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "name": "f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "status": "Failed",
      "startTime": "2026-10-08T14:28:38.920968051Z",
      "endTime": "2026-10-08T14:57:44.886200601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2a3a9b00e6d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hxkght4ww89l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "name": "f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "status": "Failed",
      "startTime": "2026-10-08T14:28:38.920968051Z",
      "endTime": "2026-10-08T14:57:44.886200601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs"
Unexpected error:
    <*fmt.wrapError | 0xa91b8f312a0>: 
    failed to create NodePool np-c7f6vs: failed waiting for nodepool="np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs" in resourcegroup="rg-candidate-5-0-mlpng9ckwtsq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79",
      "name": "630087f9-3e91-4cc3-95cc-3a0875463d79",
      "status": "Failed",
      "startTime": "2026-10-08T14:56:46.681105188Z",
      "endTime": "2026-10-08T15:25:55.347852762Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs"
Unexpected error:
    <*fmt.wrapError | 0xa91b8f312a0>: 
    failed to create NodePool np-c7f6vs: failed waiting for nodepool="np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs" in resourcegroup="rg-candidate-5-0-mlpng9ckwtsq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79",
      "name": "630087f9-3e91-4cc3-95cc-3a0875463d79",
      "status": "Failed",
      "startTime": "2026-10-08T14:56:46.681105188Z",
      "endTime": "2026-10-08T15:25:55.347852762Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n"
Unexpected error:
    <*fmt.wrapError | 0x19e2b2505020>: 
    failed to create NodePool np-cmn25n: failed waiting for nodepool="np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n" in resourcegroup="rg-candidate-5-0-lhc5fj69t9pm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080",
      "name": "6879bb7c-c254-4641-ba70-7b41bb04b080",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:10.585592043Z",
      "endTime": "2026-10-08T05:21:19.889088931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n"
Unexpected error:
    <*fmt.wrapError | 0x19e2b2505020>: 
    failed to create NodePool np-cmn25n: failed waiting for nodepool="np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n" in resourcegroup="rg-candidate-5-0-lhc5fj69t9pm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080",
      "name": "6879bb7c-c254-4641-ba70-7b41bb04b080",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:10.585592043Z",
      "endTime": "2026-10-08T05:21:19.889088931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1f97b95fe138>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zl88ncfz4jp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "name": "4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "status": "Failed",
      "startTime": "2026-10-08T04:18:50.555017189Z",
      "endTime": "2026-10-08T04:47:59.745762657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-n67fb-47fdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-n67fb-47fdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1f97b95fe138>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zl88ncfz4jp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "name": "4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "status": "Failed",
      "startTime": "2026-10-08T04:18:50.555017189Z",
      "endTime": "2026-10-08T04:47:59.745762657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-n67fb-47fdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-n67fb-47fdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x228c1da4a2e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-tff6vqrfwntq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be",
      "name": "40cea923-3d28-46b1-b377-958e6c7d32be",
      "status": "Failed",
      "startTime": "2026-10-08T04:17:53.289622119Z",
      "endTime": "2026-10-08T04:46:59.632942955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-7vfqr-gvgx9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-7vfqr-gvgx9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x228c1da4a2e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-tff6vqrfwntq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be",
      "name": "40cea923-3d28-46b1-b377-958e6c7d32be",
      "status": "Failed",
      "startTime": "2026-10-08T04:17:53.289622119Z",
      "endTime": "2026-10-08T04:46:59.632942955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-7vfqr-gvgx9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-7vfqr-gvgx9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xbf2bf37c078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rgbgvj2hmvgv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "name": "5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "status": "Failed",
      "startTime": "2026-10-08T04:21:09.226625204Z",
      "endTime": "2026-10-08T04:50:09.941476385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xbf2bf37c078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rgbgvj2hmvgv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "name": "5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "status": "Failed",
      "startTime": "2026-10-08T04:21:09.226625204Z",
      "endTime": "2026-10-08T04:50:09.941476385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9"
Unexpected error:
    <*fmt.wrapError | 0x28a272d90ba0>: 
    failed to create NodePool np-2wf5g9: failed waiting for nodepool="np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9" in resourcegroup="rg-candidate-5-1-tpzmdkt924rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "name": "acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:41.296917226Z",
      "endTime": "2026-10-08T05:21:49.770952812Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9"
Unexpected error:
    <*fmt.wrapError | 0x28a272d90ba0>: 
    failed to create NodePool np-2wf5g9: failed waiting for nodepool="np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9" in resourcegroup="rg-candidate-5-1-tpzmdkt924rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "name": "acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:41.296917226Z",
      "endTime": "2026-10-08T05:21:49.770952812Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5fp562" for cluster "cluster-candidate-4-22-5fp562"
Unexpected error:
    <*fmt.wrapError | 0x1acc4afec200>: 
    failed to create NodePool np-5fp562: failed waiting for nodepool="np-5fp562" for cluster "cluster-candidate-4-22-5fp562" in resourcegroup="rg-candidate-4-22-x7bk4qfg72wt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "name": "df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "status": "Failed",
      "startTime": "2026-10-08T04:49:21.745083143Z",
      "endTime": "2026-10-08T05:18:29.866533862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5fp562" for cluster "cluster-candidate-4-22-5fp562"
Unexpected error:
    <*fmt.wrapError | 0x1acc4afec200>: 
    failed to create NodePool np-5fp562: failed waiting for nodepool="np-5fp562" for cluster "cluster-candidate-4-22-5fp562" in resourcegroup="rg-candidate-4-22-x7bk4qfg72wt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "name": "df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "status": "Failed",
      "startTime": "2026-10-08T04:49:21.745083143Z",
      "endTime": "2026-10-08T05:18:29.866533862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x12d67242a00>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdb7hgvdsm8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "name": "1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "status": "Failed",
      "startTime": "2026-10-08T05:42:55.903350034Z",
      "endTime": "2026-10-08T06:12:00.742498982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x12d67242a00>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdb7hgvdsm8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "name": "1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "status": "Failed",
      "startTime": "2026-10-08T05:42:55.903350034Z",
      "endTime": "2026-10-08T06:12:00.742498982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3cad2d10c0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4xktww2hxb7t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "name": "7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "status": "Failed",
      "startTime": "2026-10-08T04:19:51.649780345Z",
      "endTime": "2026-10-08T04:48:59.779189852Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3cad2d10c0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4xktww2hxb7t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "name": "7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "status": "Failed",
      "startTime": "2026-10-08T04:19:51.649780345Z",
      "endTime": "2026-10-08T04:48:59.779189852Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4"
Unexpected error:
    <*fmt.wrapError | 0x1e5feac2cf20>: 
    failed to create NodePool np-nm4ft4: failed waiting for nodepool="np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4" in resourcegroup="rg-candidate-4-20-gwp4bdzvdkfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65",
      "name": "7d0b1c33-9150-4065-8244-327119876e65",
      "status": "Failed",
      "startTime": "2026-10-08T04:47:03.009170999Z",
      "endTime": "2026-10-08T05:16:09.831739591Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4"
Unexpected error:
    <*fmt.wrapError | 0x1e5feac2cf20>: 
    failed to create NodePool np-nm4ft4: failed waiting for nodepool="np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4" in resourcegroup="rg-candidate-4-20-gwp4bdzvdkfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65",
      "name": "7d0b1c33-9150-4065-8244-327119876e65",
      "status": "Failed",
      "startTime": "2026-10-08T04:47:03.009170999Z",
      "endTime": "2026-10-08T05:16:09.831739591Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn"
Unexpected error:
    <*fmt.wrapError | 0x27ed88bbf980>: 
    failed to create NodePool np-kjsxrn: failed waiting for nodepool="np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn" in resourcegroup="rg-candidate-4-23-9blkbrrd87j2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51",
      "name": "ea16240d-6235-4ea8-8e63-3873031d6d51",
      "status": "Failed",
      "startTime": "2026-10-08T04:54:34.518182875Z",
      "endTime": "2026-10-08T05:23:35.635646755Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn"
Unexpected error:
    <*fmt.wrapError | 0x27ed88bbf980>: 
    failed to create NodePool np-kjsxrn: failed waiting for nodepool="np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn" in resourcegroup="rg-candidate-4-23-9blkbrrd87j2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51",
      "name": "ea16240d-6235-4ea8-8e63-3873031d6d51",
      "status": "Failed",
      "startTime": "2026-10-08T04:54:34.518182875Z",
      "endTime": "2026-10-08T05:23:35.635646755Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c"
Unexpected error:
    <*fmt.wrapError | 0x39a1a2b36a20>: 
    failed to create NodePool np-5bpm5c: failed waiting for nodepool="np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c" in resourcegroup="rg-candidate-4-21-vptzpz86kwlt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226",
      "name": "5b709679-17d4-47e9-a453-c0f7086a9226",
      "status": "Failed",
      "startTime": "2026-10-08T04:50:41.026700325Z",
      "endTime": "2026-10-08T05:19:49.834642675Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c"
Unexpected error:
    <*fmt.wrapError | 0x39a1a2b36a20>: 
    failed to create NodePool np-5bpm5c: failed waiting for nodepool="np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c" in resourcegroup="rg-candidate-4-21-vptzpz86kwlt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226",
      "name": "5b709679-17d4-47e9-a453-c0f7086a9226",
      "status": "Failed",
      "startTime": "2026-10-08T04:50:41.026700325Z",
      "endTime": "2026-10-08T05:19:49.834642675Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x3bcb55dd85a0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-crphwg" resourcegroup="rg-np-dg-minor-crphwg-6m9n6b7fcg8d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-crphwg-6m9n6b7fcg8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-crphwg/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x3bcb55dd85a0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-crphwg" resourcegroup="rg-np-dg-minor-crphwg-6m9n6b7fcg8d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-crphwg-6m9n6b7fcg8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-crphwg/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt"
Unexpected error:
    <*fmt.wrapError | 0x23e3b649efa0>: 
    failed to create NodePool np-8tzhmt: failed waiting for nodepool="np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt" in resourcegroup="rg-candidate-4-22-sbbrjn549jd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75",
      "name": "33868d99-b424-4bf0-8b0a-debdff616e75",
      "status": "Failed",
      "startTime": "2026-10-08T02:47:01.121478067Z",
      "endTime": "2026-10-08T03:16:05.005157282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt"
Unexpected error:
    <*fmt.wrapError | 0x23e3b649efa0>: 
    failed to create NodePool np-8tzhmt: failed waiting for nodepool="np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt" in resourcegroup="rg-candidate-4-22-sbbrjn549jd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75",
      "name": "33868d99-b424-4bf0-8b0a-debdff616e75",
      "status": "Failed",
      "startTime": "2026-10-08T02:47:01.121478067Z",
      "endTime": "2026-10-08T03:16:05.005157282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg"
Unexpected error:
    <*fmt.wrapError | 0x1052ae90d2e0>: 
    failed to create NodePool np-wlzmmg: failed waiting for nodepool="np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg" in resourcegroup="rg-candidate-5-1-2rb9r54kz678" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c",
      "name": "ba8e191c-888b-485b-8866-d49e0334cc7c",
      "status": "Failed",
      "startTime": "2026-10-08T02:51:25.134077752Z",
      "endTime": "2026-10-08T03:20:27.169387826Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg"
Unexpected error:
    <*fmt.wrapError | 0x1052ae90d2e0>: 
    failed to create NodePool np-wlzmmg: failed waiting for nodepool="np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg" in resourcegroup="rg-candidate-5-1-2rb9r54kz678" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c",
      "name": "ba8e191c-888b-485b-8866-d49e0334cc7c",
      "status": "Failed",
      "startTime": "2026-10-08T02:51:25.134077752Z",
      "endTime": "2026-10-08T03:20:27.169387826Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x4fa9c508420>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s9lxbp6g9wp9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x4fa9c508420>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s9lxbp6g9wp9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2523ecbc0ea0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zgfh95c7gz2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "name": "cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "status": "Failed",
      "startTime": "2026-10-08T03:28:30.373260167Z",
      "endTime": "2026-10-08T03:57:35.369426461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2523ecbc0ea0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zgfh95c7gz2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "name": "cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "status": "Failed",
      "startTime": "2026-10-08T03:28:30.373260167Z",
      "endTime": "2026-10-08T03:57:35.369426461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-l5lgrv" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x1c5666c0fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-hhxjcw4kwjpv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-l5lgrv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-l5lgrv" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x1c5666c0fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-hhxjcw4kwjpv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-l5lgrv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-bd76nk" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1ad51ba8c390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-xg2dpqfdj5bt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-bd76nk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-bd76nk" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1ad51ba8c390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-xg2dpqfdj5bt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-bd76nk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-kdl52j" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xed02447ca80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-lljlhvtp7gqk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-kdl52j
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-kdl52j" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xed02447ca80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-lljlhvtp7gqk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-kdl52j
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74"
Unexpected error:
    <*fmt.wrapError | 0xb57001bd580>: 
    failed to create NodePool np-tkcz74: failed waiting for nodepool="np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74" in resourcegroup="rg-candidate-4-21-qcxhk95mnhvk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077",
      "name": "95120e83-d81e-45c6-bda2-21b985646077",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:34.00484297Z",
      "endTime": "2026-10-08T03:14:34.986063906Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74"
Unexpected error:
    <*fmt.wrapError | 0xb57001bd580>: 
    failed to create NodePool np-tkcz74: failed waiting for nodepool="np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74" in resourcegroup="rg-candidate-4-21-qcxhk95mnhvk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077",
      "name": "95120e83-d81e-45c6-bda2-21b985646077",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:34.00484297Z",
      "endTime": "2026-10-08T03:14:34.986063906Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6b03fb724a0>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-4bzrjj" resourcegroup="rg-np-noedge-4bzrjj-n97rz8gmwdvr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4bzrjj-n97rz8gmwdvr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-4bzrjj/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6b03fb724a0>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-4bzrjj" resourcegroup="rg-np-noedge-4bzrjj-n97rz8gmwdvr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4bzrjj-n97rz8gmwdvr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-4bzrjj/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0xc813599cf60>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-dz427r" resourcegroup="rg-np-skip-minor-dz427r-crp6tpsxwgq6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dz427r-crp6tpsxwgq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dz427r/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0xc813599cf60>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-dz427r" resourcegroup="rg-np-skip-minor-dz427r-crp6tpsxwgq6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dz427r-crp6tpsxwgq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dz427r/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-64zx6d" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xd58a144ed0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-dstvpqcxjmxk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-64zx6d
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-64zx6d" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xd58a144ed0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-dstvpqcxjmxk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-64zx6d
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2c220eff60c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-bxcw427xldqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "name": "f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:39.603761603Z",
      "endTime": "2026-10-08T02:47:44.845130459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2c220eff60c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-bxcw427xldqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "name": "f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:39.603761603Z",
      "endTime": "2026-10-08T02:47:44.845130459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x26e1bcba84a0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-7jfzmq" resourcegroup="rg-np-downgrade-7jfzmq-rvr22sztdn98": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-7jfzmq-rvr22sztdn98/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-7jfzmq/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x26e1bcba84a0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-7jfzmq" resourcegroup="rg-np-downgrade-7jfzmq-rvr22sztdn98": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-7jfzmq-rvr22sztdn98/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-7jfzmq/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-hn2cnv" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x9116922ea20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-4w8m65kzp5nn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-hn2cnv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-hn2cnv" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x9116922ea20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-4w8m65kzp5nn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-hn2cnv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-xjwrx8" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x3f5c96b45080>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-lt88zk4qvv8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-xjwrx8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-xjwrx8" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x3f5c96b45080>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-lt88zk4qvv8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-xjwrx8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x28f92e3a99c0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-tw9ncc" resourcegroup="rg-np-version-upgrade-tw9ncc-98zjd59jkjpr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tw9ncc-98zjd59jkjpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tw9ncc/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x28f92e3a99c0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-tw9ncc" resourcegroup="rg-np-version-upgrade-tw9ncc-98zjd59jkjpr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tw9ncc-98zjd59jkjpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tw9ncc/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x65e48efd800>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-qwp7bbpclw7g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "name": "1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "status": "Failed",
      "startTime": "2026-10-08T02:19:11.90853804Z",
      "endTime": "2026-10-08T02:48:14.711261391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x65e48efd800>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-qwp7bbpclw7g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "name": "1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "status": "Failed",
      "startTime": "2026-10-08T02:19:11.90853804Z",
      "endTime": "2026-10-08T02:48:14.711261391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-x2nmgc" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x3577745b47e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-l49w7x6sf8lk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x2nmgc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-x2nmgc" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x3577745b47e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-l49w7x6sf8lk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x2nmgc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x35b2e2ff91e0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-rx2rkxms8sw7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-rx2rkxms8sw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x35b2e2ff91e0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-rx2rkxms8sw7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-rx2rkxms8sw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-wps54w" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x3ddfbef4d200>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-9qcq2bc8pfhn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-wps54w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-wps54w" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x3ddfbef4d200>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-9qcq2bc8pfhn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-wps54w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj"
Unexpected error:
    <*fmt.wrapError | 0x17bbc4694220>: 
    failed to create NodePool np-znb5qj: failed waiting for nodepool="np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj" in resourcegroup="rg-candidate-5-0-d5trjc74qhsj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac",
      "name": "14de4838-40ca-415c-a29b-7d78f06ecdac",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:59.670336775Z",
      "endTime": "2026-10-08T03:19:07.230773535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj"
Unexpected error:
    <*fmt.wrapError | 0x17bbc4694220>: 
    failed to create NodePool np-znb5qj: failed waiting for nodepool="np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj" in resourcegroup="rg-candidate-5-0-d5trjc74qhsj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac",
      "name": "14de4838-40ca-415c-a29b-7d78f06ecdac",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:59.670336775Z",
      "endTime": "2026-10-08T03:19:07.230773535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:101]: failed to deploy dedicated managed identities
Unexpected error:
    <*fmt.wrapError | 0x2a6c3876d160>: 
    failed waiting for deployment "mi-missing-mi-hcp-cluster-8qb82d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-missing-mi-hcp-cluster-8qb82d/operationStatuses/08584101764787793835
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterNonMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:101]: failed to deploy dedicated managed identities
Unexpected error:
    <*fmt.wrapError | 0x2a6c3876d160>: 
    failed waiting for deployment "mi-missing-mi-hcp-cluster-8qb82d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-missing-mi-hcp-cluster-8qb82d/operationStatuses/08584101764787793835
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterNonMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x6f28c1d0c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vpk5q2z5574v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x6f28c1d0c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vpk5q2z5574v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x137051db4300>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-hcx5lwfp6892/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x137051db4300>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-hcx5lwfp6892/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs"
Unexpected error:
    <*fmt.wrapError | 0x31d2ef35d8e0>: 
    failed to create NodePool np-xhn5bs: failed waiting for nodepool="np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs" in resourcegroup="rg-candidate-4-23-4llsqxjhkgt4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40",
      "name": "93da1929-fde9-4a74-85a6-ad361a679f40",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:02.43694261Z",
      "endTime": "2026-10-08T03:18:07.302711356Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs"
Unexpected error:
    <*fmt.wrapError | 0x31d2ef35d8e0>: 
    failed to create NodePool np-xhn5bs: failed waiting for nodepool="np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs" in resourcegroup="rg-candidate-4-23-4llsqxjhkgt4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40",
      "name": "93da1929-fde9-4a74-85a6-ad361a679f40",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:02.43694261Z",
      "endTime": "2026-10-08T03:18:07.302711356Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2edd5c272b00>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-df62qq" resourcegroup="rg-np-version-upgrade-df62qq-l9cc8d44b2p9": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-df62qq-l9cc8d44b2p9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-df62qq/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2edd5c272b00>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-df62qq" resourcegroup="rg-np-version-upgrade-df62qq-l9cc8d44b2p9": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-df62qq-l9cc8d44b2p9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-df62qq/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x1b63d32ea960>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-6l2qnmrqmq6s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200",
      "name": "987e3697-e775-428f-b96b-007cf98cd200",
      "status": "Failed",
      "startTime": "2026-10-08T02:17:06.289611141Z",
      "endTime": "2026-10-08T02:46:14.822515799Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-b4b48-6xc2t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-b4b48-6xc2t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x1b63d32ea960>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-6l2qnmrqmq6s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200",
      "name": "987e3697-e775-428f-b96b-007cf98cd200",
      "status": "Failed",
      "startTime": "2026-10-08T02:17:06.289611141Z",
      "endTime": "2026-10-08T02:46:14.822515799Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-b4b48-6xc2t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-b4b48-6xc2t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xf8crr" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x28f031e29260>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-mqfd4qzn4578/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xf8crr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xf8crr" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x28f031e29260>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-mqfd4qzn4578/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xf8crr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xe292679e030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fwdf5gjhw97g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499",
      "name": "e74aa6d0-c8cd-430c-8190-549622080499",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:33.237818266Z",
      "endTime": "2026-10-08T02:47:34.745399444Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xe292679e030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fwdf5gjhw97g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499",
      "name": "e74aa6d0-c8cd-430c-8190-549622080499",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:33.237818266Z",
      "endTime": "2026-10-08T02:47:34.745399444Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-pcrbcd" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x33c001cb49c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zb4tmg42vgq7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-pcrbcd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-pcrbcd" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x33c001cb49c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zb4tmg42vgq7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-pcrbcd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x19c5b50bc860>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-x2784j5xhvrh": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-x2784j5xhvrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x19c5b50bc860>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-x2784j5xhvrh": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-x2784j5xhvrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9"
Unexpected error:
    <*fmt.wrapError | 0x130c0be607c0>: 
    failed to create NodePool np-gm6fx9: failed waiting for nodepool="np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9" in resourcegroup="rg-candidate-4-20-ts84fqwf8d5v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15",
      "name": "b9435cae-155f-47ef-8d90-e886b71aff15",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:36.63094836Z",
      "endTime": "2026-10-08T03:14:44.806443674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9"
Unexpected error:
    <*fmt.wrapError | 0x130c0be607c0>: 
    failed to create NodePool np-gm6fx9: failed waiting for nodepool="np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9" in resourcegroup="rg-candidate-4-20-ts84fqwf8d5v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15",
      "name": "b9435cae-155f-47ef-8d90-e886b71aff15",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:36.63094836Z",
      "endTime": "2026-10-08T03:14:44.806443674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xa63e00795e0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-dwrcvf" resourcegroup="rg-np-version-upgrade-dwrcvf-tndq67k8xt89": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-dwrcvf-tndq67k8xt89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-dwrcvf/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xa63e00795e0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-dwrcvf" resourcegroup="rg-np-version-upgrade-dwrcvf-tndq67k8xt89": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-dwrcvf-tndq67k8xt89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-dwrcvf/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3c35adc80030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-clb6t6k9tfdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3c35adc80030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-clb6t6k9tfdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-20-g5bgdz" is viable
Unexpected error:
    <*errors.joinError | 0x2608a89861e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2608a90941e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2608a90941c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2608a81192c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2608a86f4180>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2608a86f4100>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2608a7f286e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-20-g5bgdz" is viable
Unexpected error:
    <*errors.joinError | 0x2608a89861e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2608a90941e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2608a90941c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2608a81192c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2608a86f4180>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2608a86f4100>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2608a7f286e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x226dedcde000>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-c4d5nkzpw5nz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x226dedcde000>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-c4d5nkzpw5nz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x18f7dc4001e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-v2s5z7rdtt9b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x18f7dc4001e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-v2s5z7rdtt9b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2add6a662db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-wkvrqn6clxhk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2add6a662db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-wkvrqn6clxhk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x284f14134240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sf6285dvgkzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x284f14134240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sf6285dvgkzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1480002e0090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v6mhpvmjllbc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa",
      "name": "9f4505ea-893f-413a-81c7-aa357a711eaa",
      "status": "Failed",
      "startTime": "2026-10-08T01:19:19.539417138Z",
      "endTime": "2026-10-08T01:48:27.529510506Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1480002e0090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v6mhpvmjllbc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa",
      "name": "9f4505ea-893f-413a-81c7-aa357a711eaa",
      "status": "Failed",
      "startTime": "2026-10-08T01:19:19.539417138Z",
      "endTime": "2026-10-08T01:48:27.529510506Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6"
Unexpected error:
    <*fmt.wrapError | 0xe7465c98060>: 
    failed to create NodePool np-vdrwx6: failed waiting for nodepool="np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6" in resourcegroup="rg-candidate-5-0-m92lkknqnj7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002",
      "name": "b5dc4901-b409-4643-aabf-27d9e2910002",
      "status": "Failed",
      "startTime": "2026-10-08T01:43:19.047485932Z",
      "endTime": "2026-10-08T02:12:26.297120954Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6"
Unexpected error:
    <*fmt.wrapError | 0xe7465c98060>: 
    failed to create NodePool np-vdrwx6: failed waiting for nodepool="np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6" in resourcegroup="rg-candidate-5-0-m92lkknqnj7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002",
      "name": "b5dc4901-b409-4643-aabf-27d9e2910002",
      "status": "Failed",
      "startTime": "2026-10-08T01:43:19.047485932Z",
      "endTime": "2026-10-08T02:12:26.297120954Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm"
Unexpected error:
    <*fmt.wrapError | 0x25efd83de780>: 
    failed to create NodePool np-drrgfm: failed waiting for nodepool="np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm" in resourcegroup="rg-candidate-5-1-7sfzz5wmdjlq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b",
      "name": "c439fc17-ab56-478e-a949-fbf669f84a7b",
      "status": "Failed",
      "startTime": "2026-10-08T01:47:33.608428033Z",
      "endTime": "2026-10-08T02:16:36.341701384Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm"
Unexpected error:
    <*fmt.wrapError | 0x25efd83de780>: 
    failed to create NodePool np-drrgfm: failed waiting for nodepool="np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm" in resourcegroup="rg-candidate-5-1-7sfzz5wmdjlq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b",
      "name": "c439fc17-ab56-478e-a949-fbf669f84a7b",
      "status": "Failed",
      "startTime": "2026-10-08T01:47:33.608428033Z",
      "endTime": "2026-10-08T02:16:36.341701384Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm"
Unexpected error:
    <*fmt.wrapError | 0x3576982986e0>: 
    failed to create NodePool np-pcfgpm: failed waiting for nodepool="np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm" in resourcegroup="rg-candidate-4-23-s9jhxk9d7hqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875",
      "name": "370524de-fc82-4336-87bc-65ac23926875",
      "status": "Failed",
      "startTime": "2026-10-08T01:50:33.954027938Z",
      "endTime": "2026-10-08T02:19:36.145194017Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm"
Unexpected error:
    <*fmt.wrapError | 0x3576982986e0>: 
    failed to create NodePool np-pcfgpm: failed waiting for nodepool="np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm" in resourcegroup="rg-candidate-4-23-s9jhxk9d7hqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875",
      "name": "370524de-fc82-4336-87bc-65ac23926875",
      "status": "Failed",
      "startTime": "2026-10-08T01:50:33.954027938Z",
      "endTime": "2026-10-08T02:19:36.145194017Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x86f41c2a240>: 
    resource group "rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "name": "1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:56.823039116Z",
      "endTime": "2026-10-08T01:42:56.76971285Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x86f41c2a240>: 
    resource group "rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "name": "1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:56.823039116Z",
      "endTime": "2026-10-08T01:42:56.76971285Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf"
Unexpected error:
    <*fmt.wrapError | 0x1f9795690600>: 
    failed to create NodePool np-pxdmtf: failed waiting for nodepool="np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf" in resourcegroup="rg-candidate-4-20-pb6dpm6zm8gk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "name": "7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:00.030564256Z",
      "endTime": "2026-10-08T02:11:06.112048987Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf"
Unexpected error:
    <*fmt.wrapError | 0x1f9795690600>: 
    failed to create NodePool np-pxdmtf: failed waiting for nodepool="np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf" in resourcegroup="rg-candidate-4-20-pb6dpm6zm8gk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "name": "7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:00.030564256Z",
      "endTime": "2026-10-08T02:11:06.112048987Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x170b4dc75dc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-gp2b86bd5hhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f",
      "name": "1b07d65c-7304-4507-95e3-59630e971c0f",
      "status": "Failed",
      "startTime": "2026-10-08T01:21:26.589783314Z",
      "endTime": "2026-10-08T01:50:27.09244579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x170b4dc75dc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-gp2b86bd5hhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f",
      "name": "1b07d65c-7304-4507-95e3-59630e971c0f",
      "status": "Failed",
      "startTime": "2026-10-08T01:21:26.589783314Z",
      "endTime": "2026-10-08T01:50:27.09244579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xcfd18f3b720>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r4vhjgdwsnjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "name": "d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "status": "Failed",
      "startTime": "2026-10-08T02:22:41.727850228Z",
      "endTime": "2026-10-08T02:51:47.319042789Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xcfd18f3b720>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r4vhjgdwsnjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "name": "d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "status": "Failed",
      "startTime": "2026-10-08T02:22:41.727850228Z",
      "endTime": "2026-10-08T02:51:47.319042789Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v"
Unexpected error:
    <*fmt.wrapError | 0x3d1b2ad8bcc0>: 
    failed to create NodePool np-tdmg5v: failed waiting for nodepool="np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v" in resourcegroup="rg-candidate-4-21-4tbqvrc7sgc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "name": "1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:02.12275582Z",
      "endTime": "2026-10-08T02:11:06.102672717Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v"
Unexpected error:
    <*fmt.wrapError | 0x3d1b2ad8bcc0>: 
    failed to create NodePool np-tdmg5v: failed waiting for nodepool="np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v" in resourcegroup="rg-candidate-4-21-4tbqvrc7sgc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "name": "1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:02.12275582Z",
      "endTime": "2026-10-08T02:11:06.102672717Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3c5e53f64c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p8759llzs4pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "name": "ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:57.884545997Z",
      "endTime": "2026-10-08T01:47:57.229221309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n5sb8-kxqvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n5sb8-kxqvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3c5e53f64c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p8759llzs4pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "name": "ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:57.884545997Z",
      "endTime": "2026-10-08T01:47:57.229221309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n5sb8-kxqvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n5sb8-kxqvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2"
Unexpected error:
    <*fmt.wrapError | 0xb83ba3deaa0>: 
    failed to create NodePool np-5c6cr2: failed waiting for nodepool="np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2" in resourcegroup="rg-candidate-4-22-9rjgc2ft2c5b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "name": "62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:14.829363977Z",
      "endTime": "2026-10-08T02:11:16.24601066Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2"
Unexpected error:
    <*fmt.wrapError | 0xb83ba3deaa0>: 
    failed to create NodePool np-5c6cr2: failed waiting for nodepool="np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2" in resourcegroup="rg-candidate-4-22-9rjgc2ft2c5b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "name": "62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:14.829363977Z",
      "endTime": "2026-10-08T02:11:16.24601066Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x273477d04048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zvv7lmwhhzgh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "name": "81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "status": "Failed",
      "startTime": "2026-10-08T01:30:01.277978624Z",
      "endTime": "2026-10-08T01:59:06.873419943Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x273477d04048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zvv7lmwhhzgh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "name": "81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "status": "Failed",
      "startTime": "2026-10-08T01:30:01.277978624Z",
      "endTime": "2026-10-08T01:59:06.873419943Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred