CIHealth
<
Single Day: Oct 12▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 9m 22s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 88f953d
 westus3 e2e failed #7397 (open) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
20:27:46 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x10139db40a0>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-v52tck7hh95p" to finish creating: GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "name": "567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "status": "Failed",
      "startTime": "2026-10-12T21:36:04.986032041Z",
      "endTime": "2026-10-12T21:36:44.722884546Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-v52tck7hh95p/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'ccbf438f-08ec-4e2b-ae4e-219ce7dc3762' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
            correlation ID: 7bddac41-f6a3-4695-b95b-13972ea0840e
            client request ID: ab9e631f-f159-4b76-a0df-2b299f43d7d8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x10139db40a0>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-v52tck7hh95p" to finish creating: GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "name": "567cdaaa-ef82-48d8-9481-2e386ea53f4d",
      "status": "Failed",
      "startTime": "2026-10-12T21:36:04.986032041Z",
      "endTime": "2026-10-12T21:36:44.722884546Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-v52tck7hh95p/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'ccbf438f-08ec-4e2b-ae4e-219ce7dc3762' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8314496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/567cdaaa-ef82-48d8-9481-2e386ea53f4d
            correlation ID: 7bddac41-f6a3-4695-b95b-13972ea0840e
            client request ID: ab9e631f-f159-4b76-a0df-2b299f43d7d8
occurred
2h 18m 7s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f9f9f54
 westus3 alert failed #6920 (open) 1
alert (1)
alert [hcp] KubeDeploymentReplicasMismatch fired
20:19:29 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:06:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2519552-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2519552/providers/microsoft.monitor/accounts/hcps-j2519552", namespace="ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nbzvl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T22:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2519552-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2519552/providers/microsoft.monitor/accounts/hcps-j2519552", namespace="ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-nbzvl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tdfrhqfhgs9sdc5d6upkckvedq8cool-r5h5j4f0m7l3f4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
2h 48m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head d745401
 westus3 alert failed #7472 (open) 3
alert (3)
alert [svc] BackendControllerQueueDepthHigh fired
19:36:16 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T20:59:27Z
  Ended:       2026-10-12T21:19:23Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6043392-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert [hcp] HCPClusterOperatorUnavailable fired
19:36:16 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T21:30:33Z
  Ended:       2026-10-12T22:07:32Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j6043392-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdf8lncsu04rjm64fbreumbi4l47bg3-l5g9b5r3v2p1b0s", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tdf8lncsu04rjm64fbreumbi4l47bg3-l5g9b5r3v2p1b0s (management cluster ci01-j6043392-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
alert [hcp] KubePodNotReady fired
19:36:16 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T20:51:04Z
  Ended:       2026-10-12T20:57:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6043392-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdfeve1hnh3ip3571ig9kpg8ogs5h0t-r4w0d6n8e5u2v7j", pod="etcd-2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tdfeve1hnh3ip3571ig9kpg8ogs5h0t-r4w0d6n8e5u2v7j/etcd-2 has been in a non-ready state for longer than 5 minutes.
22m 45s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f9f9f54
 centralus alert failed #6920 (open) 3
alert (3)
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
19:27:20 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2837760' could not be found."
  }
}
--------------------------------------------------------------------------------
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
19:27:20 UTC · failed at alert · test [aro-hcp-observability] [infra] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2837760' could not be found."
  }
}
--------------------------------------------------------------------------------
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
19:27:20 UTC · failed at alert · test [aro-hcp-observability] [svc] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2837760/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2837760' could not be found."
  }
}
--------------------------------------------------------------------------------
2h 24m 44s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 22fe80c
 westus3 n/a passed #7481 (open) 0 n/a
2h 13m 35s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0b9667b
 centralus alert, e2e failed ★#7479 (open) 4
e2e (2)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
18:38:25 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xe01e7de22c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2x2v84z9v765" to finish creating: GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565",
      "name": "3d382676-ff60-4e1f-babd-b92045c21565",
      "status": "Failed",
      "startTime": "2026-10-12T19:22:34.907448575Z",
      "endTime": "2026-10-12T19:51:42.771361832Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
            correlation ID: 7441ce1a-f7e5-42c1-bf5b-99ce537aeb48
            client request ID: fed3af2b-11b1-43b5-8577-26857bd2a8c1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xe01e7de22c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2x2v84z9v765" to finish creating: GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565",
      "name": "3d382676-ff60-4e1f-babd-b92045c21565",
      "status": "Failed",
      "startTime": "2026-10-12T19:22:34.907448575Z",
      "endTime": "2026-10-12T19:51:42.771361832Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d382676-ff60-4e1f-babd-b92045c21565
            correlation ID: 7441ce1a-f7e5-42c1-bf5b-99ce537aeb48
            client request ID: fed3af2b-11b1-43b5-8577-26857bd2a8c1
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-<id> to exist; provider Microsoft.RedHatOpenShift
18:38:25 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x194566f239e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-smsvt85kmkqr" to finish creating: GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0da32617-05fc-4b99-8fa9-4a08b5bd5119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0da32617-05fc-4b99-8fa9-4a08b5bd5119",
      "name": "0da32617-05fc-4b99-8fa9-4a08b5bd5119",
      "status": "Failed",
      "startTime": "2026-10-12T19:35:51.873467614Z",
      "endTime": "2026-10-12T20:04:52.500489339Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-4gnlk-rqpw4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-smsvt85kmkqr--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-4gnlk-rqpw4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-4gnlk-rqpw4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-smsvt85kmkqr--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-4gnlk-rqpw4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0da32617-05fc-4b99-8fa9-4a08b5bd5119
            correlation ID: 6e6e9b92-3b22-4eef-a78b-96ae21c44dbf
            client request ID: d66a3161-f2e3-4d33-bb73-476a2a9b1435
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x194566f239e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-smsvt85kmkqr" to finish creating: GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0da32617-05fc-4b99-8fa9-4a08b5bd5119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0da32617-05fc-4b99-8fa9-4a08b5bd5119",
      "name": "0da32617-05fc-4b99-8fa9-4a08b5bd5119",
      "status": "Failed",
      "startTime": "2026-10-12T19:35:51.873467614Z",
      "endTime": "2026-10-12T20:04:52.500489339Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-4gnlk-rqpw4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-smsvt85kmkqr--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-4gnlk-rqpw4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-4gnlk-rqpw4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-smsvt85kmkqr--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-4gnlk-rqpw4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2661632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0da32617-05fc-4b99-8fa9-4a08b5bd5119
            correlation ID: 6e6e9b92-3b22-4eef-a78b-96ae21c44dbf
            client request ID: d66a3161-f2e3-4d33-bb73-476a2a9b1435
occurred
alert (2)
alert [hcp] KubePodCrashLooping fired
18:38:25 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:45:53Z
  Ended:       2026-10-12T20:13:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9ddee187-d906-4221-b48f-73e1102c7b1e"
  Description: Pod ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-12T19:45:53Z
  Ended:       2026-10-12T20:13:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9ddee187-d906-4221-b48f-73e1102c7b1e"
  Description: Pod ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubeStatefulSetReplicasMismatch fired
18:38:25 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:53:49Z
  Ended:       2026-10-12T20:09:48Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cdb52", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T19:53:49Z
  Ended:       2026-10-12T20:13:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2661632-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.146:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2661632/providers/microsoft.monitor/accounts/hcps-j2661632", namespace="ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cdb52", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tdeaut6dntudqr0or9vd7b3n8j9vs0m-p6q2s1e0g0h0f7y/etcd has not matched the expected number of replicas for longer than 15 minutes.
2h 27m 50s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head abae04f
 centralus e2e failed ★#7177 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
18:16:39 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-d4z57m" for cluster "cluster-hs-pre-d4z57m"
Unexpected error:
    <*fmt.wrapError | 0x3d62ef202100>: 
    failed to create NodePool np-d4z57m: failed waiting for nodepool="np-d4z57m" for cluster "cluster-hs-pre-d4z57m" in resourcegroup="rg-hs-pre--zplpn9jd4vzj" to finish creating: GET https://rp.j2582016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2867897e-0970-460c-81d6-4c5fa1ec8583
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2867897e-0970-460c-81d6-4c5fa1ec8583",
      "name": "2867897e-0970-460c-81d6-4c5fa1ec8583",
      "status": "Failed",
      "startTime": "2026-10-12T19:39:38.582635827Z",
      "endTime": "2026-10-12T20:08:43.787852166Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zplpn9jd4vzj-d4z57m--managed/providers/Microsoft.Compute/virtualMachines/f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zplpn9jd4vzj-d4z57m--managed/providers/Microsoft.Compute/virtualMachines/f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2582016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2867897e-0970-460c-81d6-4c5fa1ec8583
            correlation ID: 56d0624b-f6fa-49bb-b96e-3fa32af8a523
            client request ID: c9d4e044-239c-4fd2-8127-3707455ec72b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-d4z57m" for cluster "cluster-hs-pre-d4z57m"
Unexpected error:
    <*fmt.wrapError | 0x3d62ef202100>: 
    failed to create NodePool np-d4z57m: failed waiting for nodepool="np-d4z57m" for cluster "cluster-hs-pre-d4z57m" in resourcegroup="rg-hs-pre--zplpn9jd4vzj" to finish creating: GET https://rp.j2582016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2867897e-0970-460c-81d6-4c5fa1ec8583
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2867897e-0970-460c-81d6-4c5fa1ec8583",
      "name": "2867897e-0970-460c-81d6-4c5fa1ec8583",
      "status": "Failed",
      "startTime": "2026-10-12T19:39:38.582635827Z",
      "endTime": "2026-10-12T20:08:43.787852166Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zplpn9jd4vzj-d4z57m--managed/providers/Microsoft.Compute/virtualMachines/f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zplpn9jd4vzj-d4z57m--managed/providers/Microsoft.Compute/virtualMachines/f9u7j8t0x8a4t1u-np-d4z57m-s74xc-h7jlv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2582016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2867897e-0970-460c-81d6-4c5fa1ec8583
            correlation ID: 56d0624b-f6fa-49bb-b96e-3fa32af8a523
            client request ID: c9d4e044-239c-4fd2-8127-3707455ec72b
occurred
2h 32m 1s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5051cc7
 centralus e2e failed #7373 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
18:11:33 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-kxb95g" for cluster "cluster-hs-pre-kxb95g"
Unexpected error:
    <*fmt.wrapError | 0xb6d00db1440>: 
    failed to create NodePool np-kxb95g: failed waiting for nodepool="np-kxb95g" for cluster "cluster-hs-pre-kxb95g" in resourcegroup="rg-hs-pre--x6fpqw6hqtjb" to finish creating: GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "name": "dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "status": "Failed",
      "startTime": "2026-10-12T19:32:07.510834189Z",
      "endTime": "2026-10-12T20:01:15.571433072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
            correlation ID: bfbdef2c-b02a-48b8-a68a-a2439acfc443
            client request ID: 7e134c20-557c-48a5-9100-cbcf934467d5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-kxb95g" for cluster "cluster-hs-pre-kxb95g"
Unexpected error:
    <*fmt.wrapError | 0xb6d00db1440>: 
    failed to create NodePool np-kxb95g: failed waiting for nodepool="np-kxb95g" for cluster "cluster-hs-pre-kxb95g" in resourcegroup="rg-hs-pre--x6fpqw6hqtjb" to finish creating: GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "name": "dd473c31-c8c5-442b-a6d6-e05341bd54a5",
      "status": "Failed",
      "startTime": "2026-10-12T19:32:07.510834189Z",
      "endTime": "2026-10-12T20:01:15.571433072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2342656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/dd473c31-c8c5-442b-a6d6-e05341bd54a5
            correlation ID: bfbdef2c-b02a-48b8-a68a-a2439acfc443
            client request ID: 7e134c20-557c-48a5-9100-cbcf934467d5
occurred
2h 41m 57s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5f945eb
 centralus alert failed #7470 (open) 1
alert (1)
alert [hcp] HCPClusterOperatorUnavailable fired
18:01:54 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:50:18Z
  Ended:       2026-10-12T20:25:22Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2503936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tddoajh9tanmcldflt6gegfe8au9a4r-a8g2t7k0z6k5s9l", region="centralus", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tddoajh9tanmcldflt6gegfe8au9a4r-a8g2t7k0z6k5s9l (management cluster ci01-j2503936-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
2h 30m 47s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head e07c83d
 westus3 alert failed #7471 (open) 1
alert (1)
alert [hcp] HCPClusterOperatorUnavailable fired
17:58:00 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T19:45:35Z
  Ended:       2026-10-12T20:14:37Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j5649024-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tddn1vgtlc9nve2f6icfn7fqenmjbmm-x2t6l2r1k1q6d2z", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tddn1vgtlc9nve2f6icfn7fqenmjbmm-x2t6l2r1k1q6d2z (management cluster ci01-j5649024-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
2h 26m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 395ae23
 centralus e2e failed ⚠#7467 (open) 3
e2e (3)
ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '<timestamp>' with correlationId '<uuid>', and will expire at '<timestamp>' if it does not complete before then.
17:55:03 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:88]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xd4464611460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-b4xvfj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"DeploymentActive\",\r\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2hklr4tmfmv6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/12/2026 6:44:17 PM' with correlationId '483786d1-9a79-473c-9660-1216b08e7f90', and will expire at '10/19/2026 6:44:17 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
            correlation ID: d9670e5b-9049-4df2-b486-a7bed21f23b8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:88]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xd4464611460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-b4xvfj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"DeploymentActive\",\r\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2hklr4tmfmv6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/12/2026 6:44:17 PM' with correlationId '483786d1-9a79-473c-9660-1216b08e7f90', and will expire at '10/19/2026 6:44:17 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
            correlation ID: d9670e5b-9049-4df2-b486-a7bed21f23b8
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
17:55:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x36d0cb4c81e0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-z7qkjc46tf97" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d6731312-0cc9-4e1d-a296-bfc0d751168f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d6731312-0cc9-4e1d-a296-bfc0d751168f",
      "name": "d6731312-0cc9-4e1d-a296-bfc0d751168f",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:24.479848853Z",
      "endTime": "2026-10-12T19:14:30.222274838Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 19m29s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d6731312-0cc9-4e1d-a296-bfc0d751168f
            correlation ID: 4572b63a-1060-450d-b5db-9eae5d0fdb78
            client request ID: 5f229b06-fd04-479f-bdad-788fe5458949
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x36d0cb4c81e0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-z7qkjc46tf97" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d6731312-0cc9-4e1d-a296-bfc0d751168f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d6731312-0cc9-4e1d-a296-bfc0d751168f",
      "name": "d6731312-0cc9-4e1d-a296-bfc0d751168f",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:24.479848853Z",
      "endTime": "2026-10-12T19:14:30.222274838Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 19m29s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d6731312-0cc9-4e1d-a296-bfc0d751168f
            correlation ID: 4572b63a-1060-450d-b5db-9eae5d0fdb78
            client request ID: 5f229b06-fd04-479f-bdad-788fe5458949
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
17:55:03 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0xe13d9ba2880>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-9czhr6hdgkds" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "name": "abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:22.752877991Z",
      "endTime": "2026-10-12T19:14:30.175613398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
            correlation ID: e99f3924-5824-4d14-bfb3-a5c2c635c6fb
            client request ID: 3e3fb9a9-7a4c-4e3b-9ae6-64fae2a5931a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0xe13d9ba2880>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-9czhr6hdgkds" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "name": "abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:22.752877991Z",
      "endTime": "2026-10-12T19:14:30.175613398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
            correlation ID: e99f3924-5824-4d14-bfb3-a5c2c635c6fb
            client request ID: 3e3fb9a9-7a4c-4e3b-9ae6-64fae2a5931a
occurred
2h 41m 55s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 209c55b
 centralus n/a passed #7468 (open) 0 n/a
2h 10m 15s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 521d3fc
 centralus n/a passed #7376 (open) 0 n/a
2h 14m 37s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5cffec4 · merge 5cffec4
 westus3 n/a passed #7478 (merged) 0 n/a
2h 32m 14s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 21c5097
 centralus e2e failed ⚠#7162 (open) 2
e2e (2)
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36
16:01:28 UTC · failed at e2e · test SRE advances a control plane to an SRE-pinned version · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-zx5s4v" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-zx5s4v" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
16:01:28 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xb019e0de2e8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xb019c86fe00>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c86fda0>{
                    msg: "failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df811d0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xb019c4d6ff0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019bfab1e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019bfab0e0>{
                    msg: "failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df140c0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xb019c5e2a50>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019c4e0980>{
                msg: "VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c4e0960>{
                    msg: "failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019e2caff0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default",
                        Err: <*net.OpError | 0xb019bfc47d0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xb019e0de2e8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xb019c86fe00>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c86fda0>{
                    msg: "failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df811d0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xb019c4d6ff0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019bfab1e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019bfab0e0>{
                    msg: "failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df140c0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xb019c5e2a50>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019c4e0980>{
                msg: "VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c4e0960>{
                    msg: "failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019e2caff0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default",
                        Err: <*net.OpError | 0xb019bfc47d0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
2h 19m 41s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f71a5d1
 westus3 alert, e2e failed ⚠#7477 (open) 53
e2e (48)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-zf2xkx" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x20d83d0b6300>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-zf2xkx: failed waiting for cluster="cp-ystream-upgrade-4-22-zf2xkx" in resourcegroup="rg-cp-ystream-upgrade-4-22-z4gr6cw446tn" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a0ccc8b-6d58-40e9-9350-d1b7cc14d624
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a0ccc8b-6d58-40e9-9350-d1b7cc14d624",
      "name": "4a0ccc8b-6d58-40e9-9350-d1b7cc14d624",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:51.296200468Z",
      "endTime": "2026-10-12T16:33:59.904433857Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a0ccc8b-6d58-40e9-9350-d1b7cc14d624
            correlation ID: 533c542a-908f-40a3-ad31-456c1c10be02
            client request ID: b534539f-cc46-4687-aa71-9ff6931d4410
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-zf2xkx" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x20d83d0b6300>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-zf2xkx: failed waiting for cluster="cp-ystream-upgrade-4-22-zf2xkx" in resourcegroup="rg-cp-ystream-upgrade-4-22-z4gr6cw446tn" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a0ccc8b-6d58-40e9-9350-d1b7cc14d624
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a0ccc8b-6d58-40e9-9350-d1b7cc14d624",
      "name": "4a0ccc8b-6d58-40e9-9350-d1b7cc14d624",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:51.296200468Z",
      "endTime": "2026-10-12T16:33:59.904433857Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a0ccc8b-6d58-40e9-9350-d1b7cc14d624
            correlation ID: 533c542a-908f-40a3-ad31-456c1c10be02
            client request ID: b534539f-cc46-4687-aa71-9ff6931d4410
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x15d41e5c6240>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-5j4jfht4qlmg" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2da82feb-2183-4a70-9d57-b0fc9cc5fb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2da82feb-2183-4a70-9d57-b0fc9cc5fb71",
      "name": "2da82feb-2183-4a70-9d57-b0fc9cc5fb71",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:19.765893273Z",
      "endTime": "2026-10-12T16:34:19.389105786Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2da82feb-2183-4a70-9d57-b0fc9cc5fb71
            correlation ID: 9be86eb4-1760-485e-aa02-0152aaa7b2dd
            client request ID: ea7e40c3-adce-40f3-9337-36426044a94c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x15d41e5c6240>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-5j4jfht4qlmg" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2da82feb-2183-4a70-9d57-b0fc9cc5fb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2da82feb-2183-4a70-9d57-b0fc9cc5fb71",
      "name": "2da82feb-2183-4a70-9d57-b0fc9cc5fb71",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:19.765893273Z",
      "endTime": "2026-10-12T16:34:19.389105786Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2da82feb-2183-4a70-9d57-b0fc9cc5fb71
            correlation ID: 9be86eb4-1760-485e-aa02-0152aaa7b2dd
            client request ID: ea7e40c3-adce-40f3-9337-36426044a94c
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:94]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x207341b0e2c0>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-xtxmlffltb9n" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e92df330-b202-4f70-bed1-e815e94ef4f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e92df330-b202-4f70-bed1-e815e94ef4f3",
      "name": "e92df330-b202-4f70-bed1-e815e94ef4f3",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:14.427882995Z",
      "endTime": "2026-10-12T16:34:19.650879023Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e92df330-b202-4f70-bed1-e815e94ef4f3
            correlation ID: b072c371-50d5-4b18-b7e4-f792c3250769
            client request ID: 861c25eb-91cb-4175-813e-baf5e45adfb4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:94]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x207341b0e2c0>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-xtxmlffltb9n" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e92df330-b202-4f70-bed1-e815e94ef4f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e92df330-b202-4f70-bed1-e815e94ef4f3",
      "name": "e92df330-b202-4f70-bed1-e815e94ef4f3",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:14.427882995Z",
      "endTime": "2026-10-12T16:34:19.650879023Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e92df330-b202-4f70-bed1-e815e94ef4f3
            correlation ID: b072c371-50d5-4b18-b7e4-f792c3250769
            client request ID: 861c25eb-91cb-4175-813e-baf5e45adfb4
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x1042c14a00e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-2nr5sf886lw6" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dac1a74-819f-41d7-8d23-79db0084d547
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dac1a74-819f-41d7-8d23-79db0084d547",
      "name": "6dac1a74-819f-41d7-8d23-79db0084d547",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:51.742280808Z",
      "endTime": "2026-10-12T16:33:59.639369344Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dac1a74-819f-41d7-8d23-79db0084d547
            correlation ID: d7935d7f-d7f9-4e3e-9404-6468dba0cf73
            client request ID: 448194cd-f1a7-4755-85d1-3802ea5135b8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x1042c14a00e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-2nr5sf886lw6" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dac1a74-819f-41d7-8d23-79db0084d547
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dac1a74-819f-41d7-8d23-79db0084d547",
      "name": "6dac1a74-819f-41d7-8d23-79db0084d547",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:51.742280808Z",
      "endTime": "2026-10-12T16:33:59.639369344Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6dac1a74-819f-41d7-8d23-79db0084d547
            correlation ID: d7935d7f-d7f9-4e3e-9404-6468dba0cf73
            client request ID: 448194cd-f1a7-4755-85d1-3802ea5135b8
occurred
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
15:17:08 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x4c656e042c0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kbvm65vdfvv2" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f732fbd5-1dc4-4651-a8ce-9bcd3363a01f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f732fbd5-1dc4-4651-a8ce-9bcd3363a01f",
      "name": "f732fbd5-1dc4-4651-a8ce-9bcd3363a01f",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:22.138065368Z",
      "endTime": "2026-10-12T16:34:29.402080119Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f732fbd5-1dc4-4651-a8ce-9bcd3363a01f
            correlation ID: 96a806b3-e759-4bed-b150-87a5032cc614
            client request ID: e02662bb-1e5e-49c5-a613-eae526fdf63d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x4c656e042c0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kbvm65vdfvv2" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f732fbd5-1dc4-4651-a8ce-9bcd3363a01f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f732fbd5-1dc4-4651-a8ce-9bcd3363a01f",
      "name": "f732fbd5-1dc4-4651-a8ce-9bcd3363a01f",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:22.138065368Z",
      "endTime": "2026-10-12T16:34:29.402080119Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f732fbd5-1dc4-4651-a8ce-9bcd3363a01f
            correlation ID: 96a806b3-e759-4bed-b150-87a5032cc614
            client request ID: e02662bb-1e5e-49c5-a613-eae526fdf63d
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0x3319f6bce300>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-bqhxbgbszsk7" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9411d8dc-7343-4af4-8a8c-e09d46093966
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9411d8dc-7343-4af4-8a8c-e09d46093966",
      "name": "9411d8dc-7343-4af4-8a8c-e09d46093966",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.24212214Z",
      "endTime": "2026-10-12T16:34:19.71675375Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9411d8dc-7343-4af4-8a8c-e09d46093966
            correlation ID: 83c55d94-e4f5-48dc-ab1d-2435b0924cd2
            client request ID: 5a6c1344-e9f4-4002-b03a-1f3a2c281b13
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0x3319f6bce300>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-bqhxbgbszsk7" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9411d8dc-7343-4af4-8a8c-e09d46093966
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9411d8dc-7343-4af4-8a8c-e09d46093966",
      "name": "9411d8dc-7343-4af4-8a8c-e09d46093966",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.24212214Z",
      "endTime": "2026-10-12T16:34:19.71675375Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9411d8dc-7343-4af4-8a8c-e09d46093966
            correlation ID: 83c55d94-e4f5-48dc-ab1d-2435b0924cd2
            client request ID: 5a6c1344-e9f4-4002-b03a-1f3a2c281b13
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x30d9dda2acc0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-24bgqfrw95w6" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2f28c6-4bd8-443b-bc1a-dfe1125fba3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2f28c6-4bd8-443b-bc1a-dfe1125fba3d",
      "name": "de2f28c6-4bd8-443b-bc1a-dfe1125fba3d",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:32.35534835Z",
      "endTime": "2026-10-12T16:33:39.538788418Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2f28c6-4bd8-443b-bc1a-dfe1125fba3d
            correlation ID: 84c7ec2b-83b1-4fd6-bc72-b09d6c16405b
            client request ID: 260e0b34-2d09-4902-9e8e-260359ca039a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x30d9dda2acc0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-24bgqfrw95w6" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2f28c6-4bd8-443b-bc1a-dfe1125fba3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2f28c6-4bd8-443b-bc1a-dfe1125fba3d",
      "name": "de2f28c6-4bd8-443b-bc1a-dfe1125fba3d",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:32.35534835Z",
      "endTime": "2026-10-12T16:33:39.538788418Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de2f28c6-4bd8-443b-bc1a-dfe1125fba3d
            correlation ID: 84c7ec2b-83b1-4fd6-bc72-b09d6c16405b
            client request ID: 260e0b34-2d09-4902-9e8e-260359ca039a
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x26220088f600>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-m6g5ktlgc8z7" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc992b41-c033-41c6-80f2-695b7ffa7792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc992b41-c033-41c6-80f2-695b7ffa7792",
      "name": "cc992b41-c033-41c6-80f2-695b7ffa7792",
      "status": "Failed",
      "startTime": "2026-10-12T16:06:22.806272123Z",
      "endTime": "2026-10-12T16:35:29.429759823Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc992b41-c033-41c6-80f2-695b7ffa7792
            correlation ID: 2df4db99-b25a-49b2-8229-5aef1853845b
            client request ID: 7993ac50-f843-4bcf-a848-7da89c1dfb9b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x26220088f600>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-m6g5ktlgc8z7" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc992b41-c033-41c6-80f2-695b7ffa7792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc992b41-c033-41c6-80f2-695b7ffa7792",
      "name": "cc992b41-c033-41c6-80f2-695b7ffa7792",
      "status": "Failed",
      "startTime": "2026-10-12T16:06:22.806272123Z",
      "endTime": "2026-10-12T16:35:29.429759823Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc992b41-c033-41c6-80f2-695b7ffa7792
            correlation ID: 2df4db99-b25a-49b2-8229-5aef1853845b
            client request ID: 7993ac50-f843-4bcf-a848-7da89c1dfb9b
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:421]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x313891a9c3a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rc7jts7c5b7w" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86abf47d-06c9-4a9f-8086-17c447bba1ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86abf47d-06c9-4a9f-8086-17c447bba1ce",
      "name": "86abf47d-06c9-4a9f-8086-17c447bba1ce",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.029936909Z",
      "endTime": "2026-10-12T16:34:19.353815138Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86abf47d-06c9-4a9f-8086-17c447bba1ce
            correlation ID: 2dc76f4c-c923-43f3-80f7-1614d5e972ef
            client request ID: 88764da1-36ee-45ce-bf37-b0a4af6df387
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:421]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x313891a9c3a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rc7jts7c5b7w" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86abf47d-06c9-4a9f-8086-17c447bba1ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86abf47d-06c9-4a9f-8086-17c447bba1ce",
      "name": "86abf47d-06c9-4a9f-8086-17c447bba1ce",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.029936909Z",
      "endTime": "2026-10-12T16:34:19.353815138Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86abf47d-06c9-4a9f-8086-17c447bba1ce
            correlation ID: 2dc76f4c-c923-43f3-80f7-1614d5e972ef
            client request ID: 88764da1-36ee-45ce-bf37-b0a4af6df387
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x177ff7db0580>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-tpdz2pzq6nx2" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/899050a2-6fb7-479c-83ea-7740a027ce95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/899050a2-6fb7-479c-83ea-7740a027ce95",
      "name": "899050a2-6fb7-479c-83ea-7740a027ce95",
      "status": "Failed",
      "startTime": "2026-10-12T16:06:23.186199816Z",
      "endTime": "2026-10-12T16:35:29.665151108Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/899050a2-6fb7-479c-83ea-7740a027ce95
            correlation ID: 12882895-7a3c-4953-aa64-66a3a89fcf92
            client request ID: ff7030da-8a27-4cf5-a404-898aa2f8f0b7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x177ff7db0580>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-tpdz2pzq6nx2" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/899050a2-6fb7-479c-83ea-7740a027ce95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/899050a2-6fb7-479c-83ea-7740a027ce95",
      "name": "899050a2-6fb7-479c-83ea-7740a027ce95",
      "status": "Failed",
      "startTime": "2026-10-12T16:06:23.186199816Z",
      "endTime": "2026-10-12T16:35:29.665151108Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/899050a2-6fb7-479c-83ea-7740a027ce95
            correlation ID: 12882895-7a3c-4953-aa64-66a3a89fcf92
            client request ID: ff7030da-8a27-4cf5-a404-898aa2f8f0b7
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-rc5bhp" with version 4.20.41 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x1a5c448360>: 
    failed to create HCP cluster cluster-zstream-4-20-rc5bhp: failed waiting for cluster="cluster-zstream-4-20-rc5bhp" in resourcegroup="rg-zstream-upgrade-4-20-nfbcv4zwq6cm" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/590c7e12-8d58-422f-a728-bd866abfa3a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/590c7e12-8d58-422f-a728-bd866abfa3a0",
      "name": "590c7e12-8d58-422f-a728-bd866abfa3a0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:04.337525362Z",
      "endTime": "2026-10-12T16:34:09.111845276Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/590c7e12-8d58-422f-a728-bd866abfa3a0
            correlation ID: a248977c-0db4-411e-b79f-a5f86cd545ea
            client request ID: 92f67dce-a9c7-42a0-ad8e-4345b06da787
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-rc5bhp" with version 4.20.41 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x1a5c448360>: 
    failed to create HCP cluster cluster-zstream-4-20-rc5bhp: failed waiting for cluster="cluster-zstream-4-20-rc5bhp" in resourcegroup="rg-zstream-upgrade-4-20-nfbcv4zwq6cm" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/590c7e12-8d58-422f-a728-bd866abfa3a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/590c7e12-8d58-422f-a728-bd866abfa3a0",
      "name": "590c7e12-8d58-422f-a728-bd866abfa3a0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:04.337525362Z",
      "endTime": "2026-10-12T16:34:09.111845276Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/590c7e12-8d58-422f-a728-bd866abfa3a0
            correlation ID: a248977c-0db4-411e-b79f-a5f86cd545ea
            client request ID: 92f67dce-a9c7-42a0-ad8e-4345b06da787
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x2b8d1b766900>: 
    GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f36d9cc8-15b8-4ef7-9552-b1692d74dae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f36d9cc8-15b8-4ef7-9552-b1692d74dae3",
      "name": "f36d9cc8-15b8-4ef7-9552-b1692d74dae3",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:57.943824646Z",
      "endTime": "2026-10-12T16:34:59.426358813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f36d9cc8-15b8-4ef7-9552-b1692d74dae3
            correlation ID: 0b896154-4688-484a-b4e5-ac1c715565ea
            client request ID: 6c21476c-0d7a-4112-bab1-e73b18ede350
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x2b8d1b766900>: 
    GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f36d9cc8-15b8-4ef7-9552-b1692d74dae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f36d9cc8-15b8-4ef7-9552-b1692d74dae3",
      "name": "f36d9cc8-15b8-4ef7-9552-b1692d74dae3",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:57.943824646Z",
      "endTime": "2026-10-12T16:34:59.426358813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f36d9cc8-15b8-4ef7-9552-b1692d74dae3
            correlation ID: 0b896154-4688-484a-b4e5-ac1c715565ea
            client request ID: 6c21476c-0d7a-4112-bab1-e73b18ede350
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-qrw9cj with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x31c7526a720>: 
    failed to create HCP cluster np-version-upgrade-cluster-qrw9cj: failed waiting for cluster="np-version-upgrade-cluster-qrw9cj" in resourcegroup="rg-np-version-upgrade-qrw9cj-lf88x46jszn6" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc431aa9-1ae1-412b-b014-b6aeaafaa4d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc431aa9-1ae1-412b-b014-b6aeaafaa4d9",
      "name": "dc431aa9-1ae1-412b-b014-b6aeaafaa4d9",
      "status": "Failed",
      "startTime": "2026-10-12T16:32:34.07090627Z",
      "endTime": "2026-10-12T17:01:39.251657145Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc431aa9-1ae1-412b-b014-b6aeaafaa4d9
            correlation ID: bfd957bf-719e-49b3-bdac-7fd0797f19e3
            client request ID: 08772a89-5b8f-477e-91ad-296893cc14ba
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-qrw9cj with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x31c7526a720>: 
    failed to create HCP cluster np-version-upgrade-cluster-qrw9cj: failed waiting for cluster="np-version-upgrade-cluster-qrw9cj" in resourcegroup="rg-np-version-upgrade-qrw9cj-lf88x46jszn6" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc431aa9-1ae1-412b-b014-b6aeaafaa4d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc431aa9-1ae1-412b-b014-b6aeaafaa4d9",
      "name": "dc431aa9-1ae1-412b-b014-b6aeaafaa4d9",
      "status": "Failed",
      "startTime": "2026-10-12T16:32:34.07090627Z",
      "endTime": "2026-10-12T17:01:39.251657145Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc431aa9-1ae1-412b-b014-b6aeaafaa4d9
            correlation ID: bfd957bf-719e-49b3-bdac-7fd0797f19e3
            client request ID: 08772a89-5b8f-477e-91ad-296893cc14ba
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-8drrxw with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2e21426be280>: 
    failed to create HCP cluster np-version-upgrade-cluster-8drrxw: failed waiting for cluster="np-version-upgrade-cluster-8drrxw" in resourcegroup="rg-np-version-upgrade-8drrxw-2d9pwlfx6z2c" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8202470b-a319-4568-8066-47a743d0153b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8202470b-a319-4568-8066-47a743d0153b",
      "name": "8202470b-a319-4568-8066-47a743d0153b",
      "status": "Failed",
      "startTime": "2026-10-12T16:32:31.241515725Z",
      "endTime": "2026-10-12T17:01:39.223309739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8202470b-a319-4568-8066-47a743d0153b
            correlation ID: 929ee1ce-8b15-4004-9f50-2422b86c1eea
            client request ID: 92329117-f931-4d63-9f18-2bd6e17d6193
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-8drrxw with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2e21426be280>: 
    failed to create HCP cluster np-version-upgrade-cluster-8drrxw: failed waiting for cluster="np-version-upgrade-cluster-8drrxw" in resourcegroup="rg-np-version-upgrade-8drrxw-2d9pwlfx6z2c" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8202470b-a319-4568-8066-47a743d0153b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8202470b-a319-4568-8066-47a743d0153b",
      "name": "8202470b-a319-4568-8066-47a743d0153b",
      "status": "Failed",
      "startTime": "2026-10-12T16:32:31.241515725Z",
      "endTime": "2026-10-12T17:01:39.223309739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8202470b-a319-4568-8066-47a743d0153b
            correlation ID: 929ee1ce-8b15-4004-9f50-2422b86c1eea
            client request ID: 92329117-f931-4d63-9f18-2bd6e17d6193
occurred
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
15:17:08 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-xqmd9dg8sc2s/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-xqmd9dg8sc2s/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x3d33bff38b20>: 
    failed to create HCP cluster ea-listhxx92t: failed waiting for cluster="ea-listhxx92t" in resourcegroup="ea-list-6s5nhp2lwjz2" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3c17246-a7f7-45c8-8a03-331ddc91bb74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3c17246-a7f7-45c8-8a03-331ddc91bb74",
      "name": "b3c17246-a7f7-45c8-8a03-331ddc91bb74",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:25.793338411Z",
      "endTime": "2026-10-12T16:34:29.312060412Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3c17246-a7f7-45c8-8a03-331ddc91bb74
            correlation ID: c3bdb4b5-d9e1-4722-8d4b-0801c32c52a6
            client request ID: 6f856d28-2856-4395-afc1-9135fdfb41b9
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x3d33bff38b20>: 
    failed to create HCP cluster ea-listhxx92t: failed waiting for cluster="ea-listhxx92t" in resourcegroup="ea-list-6s5nhp2lwjz2" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3c17246-a7f7-45c8-8a03-331ddc91bb74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3c17246-a7f7-45c8-8a03-331ddc91bb74",
      "name": "b3c17246-a7f7-45c8-8a03-331ddc91bb74",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:25.793338411Z",
      "endTime": "2026-10-12T16:34:29.312060412Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3c17246-a7f7-45c8-8a03-331ddc91bb74
            correlation ID: c3bdb4b5-d9e1-4722-8d4b-0801c32c52a6
            client request ID: 6f856d28-2856-4395-afc1-9135fdfb41b9
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3cf76962bc0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f25877fzvj4h" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65643201-e5f5-469d-8fa3-31adca10b028
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65643201-e5f5-469d-8fa3-31adca10b028",
      "name": "65643201-e5f5-469d-8fa3-31adca10b028",
      "status": "Failed",
      "startTime": "2026-10-12T16:29:46.48300317Z",
      "endTime": "2026-10-12T16:58:49.225784039Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65643201-e5f5-469d-8fa3-31adca10b028
            correlation ID: 2d5e9971-8511-412a-9936-51793e57b7f8
            client request ID: d197ee43-d74a-43bb-ad30-5edf6b3f8b75
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3cf76962bc0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f25877fzvj4h" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65643201-e5f5-469d-8fa3-31adca10b028
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65643201-e5f5-469d-8fa3-31adca10b028",
      "name": "65643201-e5f5-469d-8fa3-31adca10b028",
      "status": "Failed",
      "startTime": "2026-10-12T16:29:46.48300317Z",
      "endTime": "2026-10-12T16:58:49.225784039Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65643201-e5f5-469d-8fa3-31adca10b028
            correlation ID: 2d5e9971-8511-412a-9936-51793e57b7f8
            client request ID: d197ee43-d74a-43bb-ad30-5edf6b3f8b75
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x36802a1a8de0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-mffj5zzpg42s" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b562f5-1652-49eb-acd3-5e2ba2ab5b2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b562f5-1652-49eb-acd3-5e2ba2ab5b2c",
      "name": "30b562f5-1652-49eb-acd3-5e2ba2ab5b2c",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:03.29403013Z",
      "endTime": "2026-10-12T16:34:09.534630695Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b562f5-1652-49eb-acd3-5e2ba2ab5b2c
            correlation ID: b0972b4f-2e18-4424-bcbb-fe9ff318739b
            client request ID: ba2567e6-2c53-40b7-88ae-12adb3090f69
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x36802a1a8de0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-mffj5zzpg42s" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b562f5-1652-49eb-acd3-5e2ba2ab5b2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b562f5-1652-49eb-acd3-5e2ba2ab5b2c",
      "name": "30b562f5-1652-49eb-acd3-5e2ba2ab5b2c",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:03.29403013Z",
      "endTime": "2026-10-12T16:34:09.534630695Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30b562f5-1652-49eb-acd3-5e2ba2ab5b2c
            correlation ID: b0972b4f-2e18-4424-bcbb-fe9ff318739b
            client request ID: ba2567e6-2c53-40b7-88ae-12adb3090f69
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0x279567c40340>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-6lf4bkr5xqfk" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85",
      "name": "2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85",
      "status": "Failed",
      "startTime": "2026-10-12T16:30:06.89817477Z",
      "endTime": "2026-10-12T16:59:09.20400123Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85
            correlation ID: 516be8ef-3151-4f6b-9ad6-def825112a39
            client request ID: 70377283-1a83-4ef0-bf32-cc22fa7b80a0
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0x279567c40340>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-6lf4bkr5xqfk" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85",
      "name": "2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85",
      "status": "Failed",
      "startTime": "2026-10-12T16:30:06.89817477Z",
      "endTime": "2026-10-12T16:59:09.20400123Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d49ef0a-2dbf-4175-b3ca-68a7b28d1c85
            correlation ID: 516be8ef-3151-4f6b-9ad6-def825112a39
            client request ID: 70377283-1a83-4ef0-bf32-cc22fa7b80a0
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-4ztzcq7pqv5d/cluster-candidate-4-21-gjxhwz should provision
Unexpected error:
    <*fmt.wrapError | 0x20aa34e90be0>: 
    failed to create HCP cluster cluster-candidate-4-21-gjxhwz: failed waiting for cluster="cluster-candidate-4-21-gjxhwz" in resourcegroup="rg-candidate-4-21-4ztzcq7pqv5d" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3692a11-8775-42df-b3ca-63fd2bc4eff6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3692a11-8775-42df-b3ca-63fd2bc4eff6",
      "name": "c3692a11-8775-42df-b3ca-63fd2bc4eff6",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.496754125Z",
      "endTime": "2026-10-12T16:34:19.708354245Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3692a11-8775-42df-b3ca-63fd2bc4eff6
            correlation ID: ac77532e-4fc3-4e55-b685-8e5b46e539cb
            client request ID: ad2be3ef-6fea-4be1-89e4-a5c81c2beb73
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-4ztzcq7pqv5d/cluster-candidate-4-21-gjxhwz should provision
Unexpected error:
    <*fmt.wrapError | 0x20aa34e90be0>: 
    failed to create HCP cluster cluster-candidate-4-21-gjxhwz: failed waiting for cluster="cluster-candidate-4-21-gjxhwz" in resourcegroup="rg-candidate-4-21-4ztzcq7pqv5d" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3692a11-8775-42df-b3ca-63fd2bc4eff6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3692a11-8775-42df-b3ca-63fd2bc4eff6",
      "name": "c3692a11-8775-42df-b3ca-63fd2bc4eff6",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.496754125Z",
      "endTime": "2026-10-12T16:34:19.708354245Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3692a11-8775-42df-b3ca-63fd2bc4eff6
            correlation ID: ac77532e-4fc3-4e55-b685-8e5b46e539cb
            client request ID: ad2be3ef-6fea-4be1-89e4-a5c81c2beb73
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xf88c4c20800>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-n8plk7fwndhf" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1fb4d6d-b979-4cd4-8635-598556293c82
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1fb4d6d-b979-4cd4-8635-598556293c82",
      "name": "a1fb4d6d-b979-4cd4-8635-598556293c82",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:28.784606371Z",
      "endTime": "2026-10-12T16:33:29.468218368Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1fb4d6d-b979-4cd4-8635-598556293c82
            correlation ID: c21803de-cafe-498c-9ca8-537b168f0464
            client request ID: e0f64d36-cf5a-405b-8e33-7500ab894b6a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xf88c4c20800>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-n8plk7fwndhf" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1fb4d6d-b979-4cd4-8635-598556293c82
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1fb4d6d-b979-4cd4-8635-598556293c82",
      "name": "a1fb4d6d-b979-4cd4-8635-598556293c82",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:28.784606371Z",
      "endTime": "2026-10-12T16:33:29.468218368Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1fb4d6d-b979-4cd4-8635-598556293c82
            correlation ID: c21803de-cafe-498c-9ca8-537b168f0464
            client request ID: e0f64d36-cf5a-405b-8e33-7500ab894b6a
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x18e96d48740>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-nmxmwv72jvsp" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/823a8af1-af0e-44ed-9ac4-a4eded671444
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/823a8af1-af0e-44ed-9ac4-a4eded671444",
      "name": "823a8af1-af0e-44ed-9ac4-a4eded671444",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.423039355Z",
      "endTime": "2026-10-12T16:34:19.36354313Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/823a8af1-af0e-44ed-9ac4-a4eded671444
            correlation ID: 8547f930-92f7-43c6-a766-b1a655689fe6
            client request ID: 4816b204-7271-40c1-94ba-c3e04d5cd401
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x18e96d48740>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-nmxmwv72jvsp" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/823a8af1-af0e-44ed-9ac4-a4eded671444
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/823a8af1-af0e-44ed-9ac4-a4eded671444",
      "name": "823a8af1-af0e-44ed-9ac4-a4eded671444",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.423039355Z",
      "endTime": "2026-10-12T16:34:19.36354313Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/823a8af1-af0e-44ed-9ac4-a4eded671444
            correlation ID: 8547f930-92f7-43c6-a766-b1a655689fe6
            client request ID: 4816b204-7271-40c1-94ba-c3e04d5cd401
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xe49980b780>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-bqcfl4xmn8nv" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca46772d-874d-464a-b1f4-1a692654e2d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca46772d-874d-464a-b1f4-1a692654e2d4",
      "name": "ca46772d-874d-464a-b1f4-1a692654e2d4",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:33.378694604Z",
      "endTime": "2026-10-12T16:33:39.38085323Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca46772d-874d-464a-b1f4-1a692654e2d4
            correlation ID: 627a4075-ce28-4086-907b-f8bcd1c50f04
            client request ID: aca0242a-0a5c-452d-9295-e9fd4a074d56
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xe49980b780>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-bqcfl4xmn8nv" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca46772d-874d-464a-b1f4-1a692654e2d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca46772d-874d-464a-b1f4-1a692654e2d4",
      "name": "ca46772d-874d-464a-b1f4-1a692654e2d4",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:33.378694604Z",
      "endTime": "2026-10-12T16:33:39.38085323Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca46772d-874d-464a-b1f4-1a692654e2d4
            correlation ID: 627a4075-ce28-4086-907b-f8bcd1c50f04
            client request ID: aca0242a-0a5c-452d-9295-e9fd4a074d56
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
15:17:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x31481464c780>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vxxh5xhm4hm8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vxxh5xhm4hm8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x314814c9ae20>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x31481464c780>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vxxh5xhm4hm8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vxxh5xhm4hm8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x314814c9ae20>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vxxh5xhm4hm8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:312]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0x12e83a5986a0>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-jqt7djmh86jz" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49dd0374-7176-4fef-8468-54280e5fe2f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49dd0374-7176-4fef-8468-54280e5fe2f0",
      "name": "49dd0374-7176-4fef-8468-54280e5fe2f0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.555692748Z",
      "endTime": "2026-10-12T16:34:19.574131662Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49dd0374-7176-4fef-8468-54280e5fe2f0
            correlation ID: ea00b3c3-aa94-4a8e-8560-6b23011adbb3
            client request ID: 1b81d0e4-7d6e-4789-ae88-b45f2e9a808d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:312]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0x12e83a5986a0>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-jqt7djmh86jz" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49dd0374-7176-4fef-8468-54280e5fe2f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49dd0374-7176-4fef-8468-54280e5fe2f0",
      "name": "49dd0374-7176-4fef-8468-54280e5fe2f0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:13.555692748Z",
      "endTime": "2026-10-12T16:34:19.574131662Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49dd0374-7176-4fef-8468-54280e5fe2f0
            correlation ID: ea00b3c3-aa94-4a8e-8560-6b23011adbb3
            client request ID: 1b81d0e4-7d6e-4789-ae88-b45f2e9a808d
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test ARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:144]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x36a986f04ae0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-ngrj2qbq5mrc" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cbadb236-9e49-4d36-8684-284ddba4ed57
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cbadb236-9e49-4d36-8684-284ddba4ed57",
      "name": "cbadb236-9e49-4d36-8684-284ddba4ed57",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:32.389348176Z",
      "endTime": "2026-10-12T16:33:39.478546952Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cbadb236-9e49-4d36-8684-284ddba4ed57
            correlation ID: cac45e57-6d25-44b2-80de-c57fabdd7948
            client request ID: 00b03fd4-4840-40a0-934f-aa0300d6a4be
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:144]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x36a986f04ae0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-ngrj2qbq5mrc" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cbadb236-9e49-4d36-8684-284ddba4ed57
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cbadb236-9e49-4d36-8684-284ddba4ed57",
      "name": "cbadb236-9e49-4d36-8684-284ddba4ed57",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:32.389348176Z",
      "endTime": "2026-10-12T16:33:39.478546952Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cbadb236-9e49-4d36-8684-284ddba4ed57
            correlation ID: cac45e57-6d25-44b2-80de-c57fabdd7948
            client request ID: 00b03fd4-4840-40a0-934f-aa0300d6a4be
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x2bb24c054ae0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-n65nc7rq5rdg" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e42e9718-8312-4170-a28d-c30c09346551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e42e9718-8312-4170-a28d-c30c09346551",
      "name": "e42e9718-8312-4170-a28d-c30c09346551",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:02.273005181Z",
      "endTime": "2026-10-12T16:34:09.206366816Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e42e9718-8312-4170-a28d-c30c09346551
            correlation ID: c61ae62a-a8ec-4dc7-bbc4-d1af2cd5934f
            client request ID: b3334a96-1e13-4945-873b-ebefe1b7128f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x2bb24c054ae0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-n65nc7rq5rdg" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e42e9718-8312-4170-a28d-c30c09346551
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e42e9718-8312-4170-a28d-c30c09346551",
      "name": "e42e9718-8312-4170-a28d-c30c09346551",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:02.273005181Z",
      "endTime": "2026-10-12T16:34:09.206366816Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e42e9718-8312-4170-a28d-c30c09346551
            correlation ID: c61ae62a-a8ec-4dc7-bbc4-d1af2cd5934f
            client request ID: b3334a96-1e13-4945-873b-ebefe1b7128f
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x18153c31e440>: 
    failed to create HCP cluster np-downgrade-mptb8x: failed waiting for cluster="np-downgrade-mptb8x" in resourcegroup="rg-np-downgrade-mptb8x-2fb42qhh2sh4" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a4ba465-c77e-48a8-b1bf-fd5ce064f81a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a4ba465-c77e-48a8-b1bf-fd5ce064f81a",
      "name": "5a4ba465-c77e-48a8-b1bf-fd5ce064f81a",
      "status": "Failed",
      "startTime": "2026-10-12T16:34:25.928266818Z",
      "endTime": "2026-10-12T17:03:29.209850456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a4ba465-c77e-48a8-b1bf-fd5ce064f81a
            correlation ID: 6cef0923-9240-4826-b9dd-aed77d039af5
            client request ID: 7bed6fac-3b1e-4c42-843d-6588be72b449
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x18153c31e440>: 
    failed to create HCP cluster np-downgrade-mptb8x: failed waiting for cluster="np-downgrade-mptb8x" in resourcegroup="rg-np-downgrade-mptb8x-2fb42qhh2sh4" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a4ba465-c77e-48a8-b1bf-fd5ce064f81a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a4ba465-c77e-48a8-b1bf-fd5ce064f81a",
      "name": "5a4ba465-c77e-48a8-b1bf-fd5ce064f81a",
      "status": "Failed",
      "startTime": "2026-10-12T16:34:25.928266818Z",
      "endTime": "2026-10-12T17:03:29.209850456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a4ba465-c77e-48a8-b1bf-fd5ce064f81a
            correlation ID: 6cef0923-9240-4826-b9dd-aed77d039af5
            client request ID: 7bed6fac-3b1e-4c42-843d-6588be72b449
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x14c939b3c680>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-c7rpzcklv7b5" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c785e249-3c26-40d1-a8a5-fad7e1a9cb69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c785e249-3c26-40d1-a8a5-fad7e1a9cb69",
      "name": "c785e249-3c26-40d1-a8a5-fad7e1a9cb69",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:52.118812309Z",
      "endTime": "2026-10-12T16:34:00.247635722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c785e249-3c26-40d1-a8a5-fad7e1a9cb69
            correlation ID: 664c91b2-48cb-49a9-9227-c75515a8f336
            client request ID: 67a94d14-b66a-48b0-a648-850ba5342d39
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x14c939b3c680>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-c7rpzcklv7b5" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c785e249-3c26-40d1-a8a5-fad7e1a9cb69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c785e249-3c26-40d1-a8a5-fad7e1a9cb69",
      "name": "c785e249-3c26-40d1-a8a5-fad7e1a9cb69",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:52.118812309Z",
      "endTime": "2026-10-12T16:34:00.247635722Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c785e249-3c26-40d1-a8a5-fad7e1a9cb69
            correlation ID: 664c91b2-48cb-49a9-9227-c75515a8f336
            client request ID: 67a94d14-b66a-48b0-a648-850ba5342d39
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x31377de45e0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sslgcqs6f9c9" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/417c11d2-2f9b-476e-aa7b-d131547953b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/417c11d2-2f9b-476e-aa7b-d131547953b6",
      "name": "417c11d2-2f9b-476e-aa7b-d131547953b6",
      "status": "Failed",
      "startTime": "2026-10-12T16:07:17.197387737Z",
      "endTime": "2026-10-12T16:36:19.074391072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/417c11d2-2f9b-476e-aa7b-d131547953b6
            correlation ID: aa74bff6-62d4-412b-8a0b-e853b2cfa1b8
            client request ID: fed0be28-4bc7-4d65-87e8-2d4e1db112b1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x31377de45e0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sslgcqs6f9c9" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/417c11d2-2f9b-476e-aa7b-d131547953b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/417c11d2-2f9b-476e-aa7b-d131547953b6",
      "name": "417c11d2-2f9b-476e-aa7b-d131547953b6",
      "status": "Failed",
      "startTime": "2026-10-12T16:07:17.197387737Z",
      "endTime": "2026-10-12T16:36:19.074391072Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/417c11d2-2f9b-476e-aa7b-d131547953b6
            correlation ID: aa74bff6-62d4-412b-8a0b-e853b2cfa1b8
            client request ID: fed0be28-4bc7-4d65-87e8-2d4e1db112b1
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--k9r29d9g92h9/cluster-hs-pre-4lqcqr should provision
Unexpected error:
    <*fmt.wrapError | 0x29b422d704c0>: 
    failed to create HCP cluster cluster-hs-pre-4lqcqr: failed waiting for cluster="cluster-hs-pre-4lqcqr" in resourcegroup="rg-hs-pre--k9r29d9g92h9" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab948028-30ab-4aa6-8829-3a06cc7c1dde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab948028-30ab-4aa6-8829-3a06cc7c1dde",
      "name": "ab948028-30ab-4aa6-8829-3a06cc7c1dde",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.199986974Z",
      "endTime": "2026-10-12T16:34:19.441219246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab948028-30ab-4aa6-8829-3a06cc7c1dde
            correlation ID: e66c132f-da52-4c57-936b-c0afdc1a831a
            client request ID: 0d1874a1-8e88-473d-8ec4-fe6ab6c34f8f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--k9r29d9g92h9/cluster-hs-pre-4lqcqr should provision
Unexpected error:
    <*fmt.wrapError | 0x29b422d704c0>: 
    failed to create HCP cluster cluster-hs-pre-4lqcqr: failed waiting for cluster="cluster-hs-pre-4lqcqr" in resourcegroup="rg-hs-pre--k9r29d9g92h9" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab948028-30ab-4aa6-8829-3a06cc7c1dde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab948028-30ab-4aa6-8829-3a06cc7c1dde",
      "name": "ab948028-30ab-4aa6-8829-3a06cc7c1dde",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.199986974Z",
      "endTime": "2026-10-12T16:34:19.441219246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab948028-30ab-4aa6-8829-3a06cc7c1dde
            correlation ID: e66c132f-da52-4c57-936b-c0afdc1a831a
            client request ID: 0d1874a1-8e88-473d-8ec4-fe6ab6c34f8f
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x495f5a50100>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-vtqpftbtl8pd" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "name": "b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.565546726Z",
      "endTime": "2026-10-12T16:34:19.299056423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
            correlation ID: 2f0e8af3-4477-4dd1-b4a2-0bb21a3ecc71
            client request ID: af43345c-178c-4f70-985a-42472759437b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x495f5a50100>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-vtqpftbtl8pd" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "name": "b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.565546726Z",
      "endTime": "2026-10-12T16:34:19.299056423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
            correlation ID: 2f0e8af3-4477-4dd1-b4a2-0bb21a3ecc71
            client request ID: af43345c-178c-4f70-985a-42472759437b
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xa246b118700>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-lv4p56sl2gn5" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a11d3580-395c-4b4e-aba8-1f155f029317
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a11d3580-395c-4b4e-aba8-1f155f029317",
      "name": "a11d3580-395c-4b4e-aba8-1f155f029317",
      "status": "Failed",
      "startTime": "2026-10-12T16:38:07.657971903Z",
      "endTime": "2026-10-12T17:07:09.209513313Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a11d3580-395c-4b4e-aba8-1f155f029317
            correlation ID: 3a5bde5c-ec4f-4189-9bf7-ce8f83338ce6
            client request ID: 628bdfcd-f4ac-4fd8-bde6-db5ebf05dcd7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xa246b118700>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-lv4p56sl2gn5" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a11d3580-395c-4b4e-aba8-1f155f029317
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a11d3580-395c-4b4e-aba8-1f155f029317",
      "name": "a11d3580-395c-4b4e-aba8-1f155f029317",
      "status": "Failed",
      "startTime": "2026-10-12T16:38:07.657971903Z",
      "endTime": "2026-10-12T17:07:09.209513313Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a11d3580-395c-4b4e-aba8-1f155f029317
            correlation ID: 3a5bde5c-ec4f-4189-9bf7-ce8f83338ce6
            client request ID: 628bdfcd-f4ac-4fd8-bde6-db5ebf05dcd7
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x50023914b40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-t9tsfck7gs88" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1269e331-6a56-4e39-8d97-876562b4239c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1269e331-6a56-4e39-8d97-876562b4239c",
      "name": "1269e331-6a56-4e39-8d97-876562b4239c",
      "status": "Failed",
      "startTime": "2026-10-12T16:30:45.872172093Z",
      "endTime": "2026-10-12T16:59:49.217488716Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1269e331-6a56-4e39-8d97-876562b4239c
            correlation ID: b81a6b2a-780b-4e63-aa63-4a20441c7bb1
            client request ID: bb9cfec1-df9f-4c19-af52-b64f941d0cea
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x50023914b40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-t9tsfck7gs88" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1269e331-6a56-4e39-8d97-876562b4239c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1269e331-6a56-4e39-8d97-876562b4239c",
      "name": "1269e331-6a56-4e39-8d97-876562b4239c",
      "status": "Failed",
      "startTime": "2026-10-12T16:30:45.872172093Z",
      "endTime": "2026-10-12T16:59:49.217488716Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1269e331-6a56-4e39-8d97-876562b4239c
            correlation ID: b81a6b2a-780b-4e63-aa63-4a20441c7bb1
            client request ID: bb9cfec1-df9f-4c19-af52-b64f941d0cea
occurred
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
15:17:08 UTC · failed at e2e · test ARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:412]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x8ceb7456d30>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:412]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x8ceb7456d30>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3390eb576ac0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-pj6qdmqzbvvk" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ca81053-a842-4371-b45d-1725686e5689
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ca81053-a842-4371-b45d-1725686e5689",
      "name": "9ca81053-a842-4371-b45d-1725686e5689",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:50.549768464Z",
      "endTime": "2026-10-12T16:34:00.22620878Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ca81053-a842-4371-b45d-1725686e5689
            correlation ID: bd5b8009-857d-40ed-a78e-2ded6716c304
            client request ID: bd2fbaf8-a68b-4aff-9d2f-f237a37aad2f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3390eb576ac0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-pj6qdmqzbvvk" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ca81053-a842-4371-b45d-1725686e5689
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ca81053-a842-4371-b45d-1725686e5689",
      "name": "9ca81053-a842-4371-b45d-1725686e5689",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:50.549768464Z",
      "endTime": "2026-10-12T16:34:00.22620878Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ca81053-a842-4371-b45d-1725686e5689
            correlation ID: bd5b8009-857d-40ed-a78e-2ded6716c304
            client request ID: bd2fbaf8-a68b-4aff-9d2f-f237a37aad2f
occurred
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
15:17:08 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0x11d59617f5d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20, targetMinor=4.21) failed: clusterversion status.history has no version in target minor "4.21"
    {
        errs: [
            <*fmt.wrapError | 0x11d596138740>{
                msg: "VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)",
                err: <*errors.errorString | 0x11d59546a3c0>{
                    s: "kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)",
                },
            },
            <*fmt.wrapError | 0x11d5961388a0>{
                msg: "VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20, targetMinor=4.21) failed: clusterversion status.history has no version in target minor \"4.21\"",
                err: <*errors.errorString | 0x11d59546a6c0>{
                    s: "clusterversion status.history has no version in target minor \"4.21\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0x11d59617f5d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20, targetMinor=4.21) failed: clusterversion status.history has no version in target minor "4.21"
    {
        errs: [
            <*fmt.wrapError | 0x11d596138740>{
                msg: "VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)",
                err: <*errors.errorString | 0x11d59546a3c0>{
                    s: "kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)",
                },
            },
            <*fmt.wrapError | 0x11d5961388a0>{
                msg: "VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20, targetMinor=4.21) failed: clusterversion status.history has no version in target minor \"4.21\"",
                err: <*errors.errorString | 0x11d59546a6c0>{
                    s: "clusterversion status.history has no version in target minor \"4.21\"",
                },
            },
        ],
    }
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0x2b5eb3004540>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-x4ktbm22q9kc" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de0f0e5b-f214-4efd-afd5-7985bdb11b91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de0f0e5b-f214-4efd-afd5-7985bdb11b91",
      "name": "de0f0e5b-f214-4efd-afd5-7985bdb11b91",
      "status": "Failed",
      "startTime": "2026-10-12T16:31:56.54495042Z",
      "endTime": "2026-10-12T17:00:59.212681854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de0f0e5b-f214-4efd-afd5-7985bdb11b91
            correlation ID: 3f6e83f7-ab63-4f4a-8e9b-190aaafac047
            client request ID: fa2bb10b-d022-4a5a-a5a4-67ed02cc2bf5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0x2b5eb3004540>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-x4ktbm22q9kc" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de0f0e5b-f214-4efd-afd5-7985bdb11b91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de0f0e5b-f214-4efd-afd5-7985bdb11b91",
      "name": "de0f0e5b-f214-4efd-afd5-7985bdb11b91",
      "status": "Failed",
      "startTime": "2026-10-12T16:31:56.54495042Z",
      "endTime": "2026-10-12T17:00:59.212681854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/de0f0e5b-f214-4efd-afd5-7985bdb11b91
            correlation ID: 3f6e83f7-ab63-4f4a-8e9b-190aaafac047
            client request ID: fa2bb10b-d022-4a5a-a5a4-67ed02cc2bf5
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:89]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x2b93509c09c0>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n45dqcs4k8ll" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f",
      "name": "f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:05.265108129Z",
      "endTime": "2026-10-12T16:34:09.122414349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f
            correlation ID: 17c8647a-3930-47ce-877f-ab9387c5b822
            client request ID: 26bd2508-f678-4251-89af-cb812e799893
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:89]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x2b93509c09c0>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-n45dqcs4k8ll" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f",
      "name": "f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:05.265108129Z",
      "endTime": "2026-10-12T16:34:09.122414349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8ace9bf-48b3-4134-a3b6-ff7e0e2f991f
            correlation ID: 17c8647a-3930-47ce-877f-ab9387c5b822
            client request ID: 26bd2508-f678-4251-89af-cb812e799893
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1c903d6aab20>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2cx7cfr7wjgf" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f3c84d-3b9d-478d-8bea-d6abeb4b5683
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f3c84d-3b9d-478d-8bea-d6abeb4b5683",
      "name": "24f3c84d-3b9d-478d-8bea-d6abeb4b5683",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:33.568522156Z",
      "endTime": "2026-10-12T16:33:39.594289919Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f3c84d-3b9d-478d-8bea-d6abeb4b5683
            correlation ID: 29f8dc5f-71ff-4129-9695-5297503fc030
            client request ID: eb69571c-de2f-4fcf-9242-baf4d64dae94
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1c903d6aab20>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2cx7cfr7wjgf" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f3c84d-3b9d-478d-8bea-d6abeb4b5683
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f3c84d-3b9d-478d-8bea-d6abeb4b5683",
      "name": "24f3c84d-3b9d-478d-8bea-d6abeb4b5683",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:33.568522156Z",
      "endTime": "2026-10-12T16:33:39.594289919Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f3c84d-3b9d-478d-8bea-d6abeb4b5683
            correlation ID: 29f8dc5f-71ff-4129-9695-5297503fc030
            client request ID: eb69571c-de2f-4fcf-9242-baf4d64dae94
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x2a02392b4920>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-cfbbzz7dz6nz" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f05566-7fcf-446c-a1f9-83457d6772d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f05566-7fcf-446c-a1f9-83457d6772d4",
      "name": "81f05566-7fcf-446c-a1f9-83457d6772d4",
      "status": "Failed",
      "startTime": "2026-10-12T16:39:24.299369974Z",
      "endTime": "2026-10-12T17:08:29.200092329Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f05566-7fcf-446c-a1f9-83457d6772d4
            correlation ID: f8f32452-a240-481d-801c-3f64f3fe12ff
            client request ID: 1ecb0ea9-2b8f-4d95-b795-94f372eb3eea
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x2a02392b4920>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-cfbbzz7dz6nz" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f05566-7fcf-446c-a1f9-83457d6772d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f05566-7fcf-446c-a1f9-83457d6772d4",
      "name": "81f05566-7fcf-446c-a1f9-83457d6772d4",
      "status": "Failed",
      "startTime": "2026-10-12T16:39:24.299369974Z",
      "endTime": "2026-10-12T17:08:29.200092329Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81f05566-7fcf-446c-a1f9-83457d6772d4
            correlation ID: f8f32452-a240-481d-801c-3f64f3fe12ff
            client request ID: 1ecb0ea9-2b8f-4d95-b795-94f372eb3eea
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x212fbc084a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-8hkmlh8qnx7w" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d399ee-b335-46b2-ac26-ebca4137d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d399ee-b335-46b2-ac26-ebca4137d374",
      "name": "a5d399ee-b335-46b2-ac26-ebca4137d374",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:53.824928754Z",
      "endTime": "2026-10-12T16:34:00.204962099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d399ee-b335-46b2-ac26-ebca4137d374
            correlation ID: 2d28ca59-634e-45e7-aa28-3231a3cd431f
            client request ID: 9ee94869-89d4-4850-9db2-977e7eaffbfa
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x212fbc084a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-8hkmlh8qnx7w" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d399ee-b335-46b2-ac26-ebca4137d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d399ee-b335-46b2-ac26-ebca4137d374",
      "name": "a5d399ee-b335-46b2-ac26-ebca4137d374",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:53.824928754Z",
      "endTime": "2026-10-12T16:34:00.204962099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d399ee-b335-46b2-ac26-ebca4137d374
            correlation ID: 2d28ca59-634e-45e7-aa28-3231a3cd431f
            client request ID: 9ee94869-89d4-4850-9db2-977e7eaffbfa
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x8d9ed1ecf40>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-7lbnvkdmwzbl" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e62beee-100e-4e5e-91cf-7bb6624ff2a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e62beee-100e-4e5e-91cf-7bb6624ff2a0",
      "name": "9e62beee-100e-4e5e-91cf-7bb6624ff2a0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:56.267179906Z",
      "endTime": "2026-10-12T16:34:59.200522398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e62beee-100e-4e5e-91cf-7bb6624ff2a0
            correlation ID: 2520b360-0d36-4ca0-bb12-eef341706d5b
            client request ID: 205b5e74-f26c-4c8a-80fe-6cd33ba159fc
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x8d9ed1ecf40>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-7lbnvkdmwzbl" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e62beee-100e-4e5e-91cf-7bb6624ff2a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e62beee-100e-4e5e-91cf-7bb6624ff2a0",
      "name": "9e62beee-100e-4e5e-91cf-7bb6624ff2a0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:56.267179906Z",
      "endTime": "2026-10-12T16:34:59.200522398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e62beee-100e-4e5e-91cf-7bb6624ff2a0
            correlation ID: 2520b360-0d36-4ca0-bb12-eef341706d5b
            client request ID: 205b5e74-f26c-4c8a-80fe-6cd33ba159fc
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Engineering want metrics from HCP to be present in Azure Monitor Workspace · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x3f41f431c220>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-94qwhdgdlfld" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc606312-a6f8-4a33-bf01-c5d936147d70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc606312-a6f8-4a33-bf01-c5d936147d70",
      "name": "bc606312-a6f8-4a33-bf01-c5d936147d70",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:15.532663446Z",
      "endTime": "2026-10-12T16:34:19.695494446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc606312-a6f8-4a33-bf01-c5d936147d70
            correlation ID: 32984428-9d46-4a6b-8619-1d1460e2ac1b
            client request ID: b088058d-962b-4eb2-9131-24b6f9ceeb11
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x3f41f431c220>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-94qwhdgdlfld" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc606312-a6f8-4a33-bf01-c5d936147d70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc606312-a6f8-4a33-bf01-c5d936147d70",
      "name": "bc606312-a6f8-4a33-bf01-c5d936147d70",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:15.532663446Z",
      "endTime": "2026-10-12T16:34:19.695494446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc606312-a6f8-4a33-bf01-c5d936147d70
            correlation ID: 32984428-9d46-4a6b-8619-1d1460e2ac1b
            client request ID: b088058d-962b-4eb2-9131-24b6f9ceeb11
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1720c4528160>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-248bgmphdnf7" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437b47ce-9c3f-4407-aeb1-fd10ad4d82e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437b47ce-9c3f-4407-aeb1-fd10ad4d82e0",
      "name": "437b47ce-9c3f-4407-aeb1-fd10ad4d82e0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.4831266Z",
      "endTime": "2026-10-12T16:34:19.148385019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437b47ce-9c3f-4407-aeb1-fd10ad4d82e0
            correlation ID: 776bda98-e7b3-4551-a040-3ba05f709a45
            client request ID: e18b8bd2-0ab9-4a4b-af54-2fdf8b5f8e56
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1720c4528160>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-248bgmphdnf7" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437b47ce-9c3f-4407-aeb1-fd10ad4d82e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437b47ce-9c3f-4407-aeb1-fd10ad4d82e0",
      "name": "437b47ce-9c3f-4407-aeb1-fd10ad4d82e0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.4831266Z",
      "endTime": "2026-10-12T16:34:19.148385019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437b47ce-9c3f-4407-aeb1-fd10ad4d82e0
            correlation ID: 776bda98-e7b3-4551-a040-3ba05f709a45
            client request ID: e18b8bd2-0ab9-4a4b-af54-2fdf8b5f8e56
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-8h9snt" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x278858e785a0>: 
    failed to create HCP cluster cluster-zstream-4-21-8h9snt: failed waiting for cluster="cluster-zstream-4-21-8h9snt" in resourcegroup="rg-zstream-upgrade-4-21-k45g2vrqv5gs" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a71975b4-9124-4e36-a3ce-5b86ec172675
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a71975b4-9124-4e36-a3ce-5b86ec172675",
      "name": "a71975b4-9124-4e36-a3ce-5b86ec172675",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:38.921451129Z",
      "endTime": "2026-10-12T16:33:39.476865521Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a71975b4-9124-4e36-a3ce-5b86ec172675
            correlation ID: f8e3e525-426e-44f2-88ff-9793dde001bf
            client request ID: 34463a57-8054-4c90-a50e-2c4ca0477092
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-8h9snt" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x278858e785a0>: 
    failed to create HCP cluster cluster-zstream-4-21-8h9snt: failed waiting for cluster="cluster-zstream-4-21-8h9snt" in resourcegroup="rg-zstream-upgrade-4-21-k45g2vrqv5gs" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a71975b4-9124-4e36-a3ce-5b86ec172675
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a71975b4-9124-4e36-a3ce-5b86ec172675",
      "name": "a71975b4-9124-4e36-a3ce-5b86ec172675",
      "status": "Failed",
      "startTime": "2026-10-12T16:04:38.921451129Z",
      "endTime": "2026-10-12T16:33:39.476865521Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a71975b4-9124-4e36-a3ce-5b86ec172675
            correlation ID: f8e3e525-426e-44f2-88ff-9793dde001bf
            client request ID: 34463a57-8054-4c90-a50e-2c4ca0477092
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
15:17:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x37d4ffd7a600>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dvr2wr9tkktb" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d18522-dc69-4e01-bb81-5507a89165e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d18522-dc69-4e01-bb81-5507a89165e0",
      "name": "b0d18522-dc69-4e01-bb81-5507a89165e0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:53.484267787Z",
      "endTime": "2026-10-12T16:34:59.431372216Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d18522-dc69-4e01-bb81-5507a89165e0
            correlation ID: 632c6de6-daf9-402e-9574-3d088d679fc4
            client request ID: 1cc71eaf-6c2d-4d17-abdc-cf5c9a38f3ec
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x37d4ffd7a600>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dvr2wr9tkktb" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d18522-dc69-4e01-bb81-5507a89165e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d18522-dc69-4e01-bb81-5507a89165e0",
      "name": "b0d18522-dc69-4e01-bb81-5507a89165e0",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:53.484267787Z",
      "endTime": "2026-10-12T16:34:59.431372216Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d18522-dc69-4e01-bb81-5507a89165e0
            correlation ID: 632c6de6-daf9-402e-9574-3d088d679fc4
            client request ID: 1cc71eaf-6c2d-4d17-abdc-cf5c9a38f3ec
occurred
alert (5)
alert [svc] BackendControllerQueueDepthHigh fired
15:17:08 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:22:14Z
  Ended:       2026-10-12T16:44:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9626368-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert [svc] userJourneyClusterProvisionErrors6h30m fired
15:17:08 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T17:06:14Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j9626368-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9626368/providers/microsoft.monitor/accounts/services-j9626368", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j9626368-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.
alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired
15:17:08 UTC · failed at alert · test [aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption High - arohcpci01-rp-j9626368 does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:24:31Z
  Ended:       2026-10-12T16:44:33Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
alert [svc] ServiceCPUDrift fired
15:17:08 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceCPUDrift does not fire · suite aro-hcp-tests
Show raw failure
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:40:12Z
  Ended:       2026-10-12T16:50:10Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9626368-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-56bb7dd747-h9s6h", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-56bb7dd747-h9s6h (namespace aro-hcp) on cluster ci01-j9626368-svc has averaged 134.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-10-12T16:40:12Z
  Ended:       2026-10-12T16:54:08Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9626368-mgmt-2", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-c6ff9b6f4-b8fh5", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-c6ff9b6f4-b8fh5 (namespace kube-applier) on cluster ci01-j9626368-mgmt-2 has averaged 150.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-10-12T16:40:12Z
  Ended:       2026-10-12T16:56:10Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9626368-mgmt-1", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-65fd757c56-d7zhs", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-65fd757c56-d7zhs (namespace kube-applier) on cluster ci01-j9626368-mgmt-1 has averaged 164.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.
alert [svc] userJourneyClusterProvisionErrors1h5m fired
15:17:08 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors1h5m does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T17:20:12Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j9626368-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9626368/providers/microsoft.monitor/accounts/services-j9626368", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j9626368-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.
7m 41s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head a6abbc2
 n/a unknown failed #7477 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 34m 18s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f3c17b0
 centralus e2e failed ⚠#7383 (open) 2
e2e (2)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift
15:06:56 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mvkgs4" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2777a0f273c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mvkgs4: failed waiting for cluster="cp-ystream-upgrade-4-22-mvkgs4" in resourcegroup="rg-cp-ystream-upgrade-4-22-blssk6qzfvws" to finish creating: GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "name": "a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:53.687204284Z",
      "endTime": "2026-10-12T16:21:54.576185121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-policy-controller, kube-controller-manager, openshift-apiserver, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, certified-operators-catalog, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
            correlation ID: f4045377-6462-4996-b798-a9fd2a096fae
            client request ID: 01fbf192-7565-417e-a2b4-3d30deb4b951
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mvkgs4" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2777a0f273c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mvkgs4: failed waiting for cluster="cp-ystream-upgrade-4-22-mvkgs4" in resourcegroup="rg-cp-ystream-upgrade-4-22-blssk6qzfvws" to finish creating: GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "name": "a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:53.687204284Z",
      "endTime": "2026-10-12T16:21:54.576185121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-policy-controller, kube-controller-manager, openshift-apiserver, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, certified-operators-catalog, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
            correlation ID: f4045377-6462-4996-b798-a9fd2a096fae
            client request ID: 01fbf192-7565-417e-a2b4-3d30deb4b951
occurred
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36
15:06:56 UTC · failed at e2e · test SRE advances a control plane to an SRE-pinned version · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.009s.
cluster "sre-version-pin-5gfztc" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.009s.
cluster "sre-version-pin-5gfztc" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
2h 37m 4s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 395ae23
 westus3 e2e failed #7467 (open) 1
e2e (1)
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
15:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23913a80e730>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23913a80e730>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
2h 13m 8s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5051cc7
 centralus alert, e2e failed ★#7373 (open) 2
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
14:59:35 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0x10bf87371b00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-w96jh79xf66q" to finish creating: GET https://rp.j0290944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1173880a-8bf8-448d-a470-259a36f4e033
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1173880a-8bf8-448d-a470-259a36f4e033",
      "name": "1173880a-8bf8-448d-a470-259a36f4e033",
      "status": "Failed",
      "startTime": "2026-10-12T16:18:06.878392832Z",
      "endTime": "2026-10-12T16:47:15.056749067Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8 to exist\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8 to exist\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0290944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1173880a-8bf8-448d-a470-259a36f4e033
            correlation ID: 685034b4-f64b-4ad2-9968-7845cc5d96c5
            client request ID: 20bd3fae-18f5-45e3-8cd8-b2f1ea3f8020
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0x10bf87371b00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-w96jh79xf66q" to finish creating: GET https://rp.j0290944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1173880a-8bf8-448d-a470-259a36f4e033
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1173880a-8bf8-448d-a470-259a36f4e033",
      "name": "1173880a-8bf8-448d-a470-259a36f4e033",
      "status": "Failed",
      "startTime": "2026-10-12T16:18:06.878392832Z",
      "endTime": "2026-10-12T16:47:15.056749067Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8 to exist\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-6t8p8 to exist\nMachine v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-w96jh79xf66q--managed/providers/Microsoft.Compute/virtualMachines/v2n6a8e9c9p6q9j-np-1-42p4c-j6mkq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0290944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1173880a-8bf8-448d-a470-259a36f4e033
            correlation ID: 685034b4-f64b-4ad2-9968-7845cc5d96c5
            client request ID: 20bd3fae-18f5-45e3-8cd8-b2f1ea3f8020
occurred
alert (1)
alert [hcp] KubePodNotReady fired
14:59:35 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T16:16:01Z
  Ended:       2026-10-12T16:22:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0290944-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdbebgjal25ks2eu7kru50siu2d9fk7-d5m6i4f8v4g8r0f", pod="kube-controller-manager-655fcf6674-rv7x7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tdbebgjal25ks2eu7kru50siu2d9fk7-d5m6i4f8v4g8r0f/kube-controller-manager-655fcf6674-rv7x7 has been in a non-ready state for longer than 5 minutes.
2h 23m 34s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0496313
 westus3 e2e failed #7451 (open) 1
e2e (1)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
14:54:37 UTC · failed at e2e · test Engineering want metrics from HCP to be present in Azure Monitor Workspace · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3aa92878a060>: 
    resource group "hcp-metrics-w7n8h467qbnb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-w7n8h467qbnb" to finish deleting: GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103",
      "name": "eb261c81-ca83-45b2-bdfd-0de40227c103",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:20.608938335Z",
      "endTime": "2026-10-12T16:22:21.07258074Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5jcbgo4cpr0su6el4fc6i0lag0dj still exists (deletion dispatched at 2026-10-12T15:58:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
            correlation ID: 9da4d8e2-ebfd-4094-bf43-c7701d90768a
            client request ID: 61ad067a-2c1b-43a6-b798-69bd268d4471
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x3aa92878a060>: 
    resource group "hcp-metrics-w7n8h467qbnb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-w7n8h467qbnb" to finish deleting: GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103",
      "name": "eb261c81-ca83-45b2-bdfd-0de40227c103",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:20.608938335Z",
      "endTime": "2026-10-12T16:22:21.07258074Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5jcbgo4cpr0su6el4fc6i0lag0dj still exists (deletion dispatched at 2026-10-12T15:58:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4566144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb261c81-ca83-45b2-bdfd-0de40227c103
            correlation ID: 9da4d8e2-ebfd-4094-bf43-c7701d90768a
            client request ID: 61ad067a-2c1b-43a6-b798-69bd268d4471
occurred
2h 23m 51s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f9f9f54
 westus3 e2e failed #6920 (open) 5
e2e (5)
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
14:53:34 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4cmzlj277gjf/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-4cmzlj277gjf/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
14:53:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x151218b082b8>: 
    resource group "rg-cluster-back-version-9554sthg8hnz": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-9554sthg8hnz", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x151216d07800>{
                msg: "resource group \"rg-cluster-back-version-9554sthg8hnz\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x151216d077a0>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x1512170c3950>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x151218b082b8>: 
    resource group "rg-cluster-back-version-9554sthg8hnz": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-9554sthg8hnz", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9554sthg8hnz" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x151216d07800>{
                msg: "resource group \"rg-cluster-back-version-9554sthg8hnz\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                err: <*fmt.wrapError | 0x151216d077a0>{
                    msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                    err: <*fmt.wrapErrors | 0x1512170c3950>{
                        msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                        errs: [
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                            <*fmt.wrapErrors | 0x1512170c3920>{
                                msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-9554sthg8hnz\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz, error: context deadline exceeded",
                                errs: [
                                    <*errors.errorString | 0x15121775bc50>{
                                        s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-9554sthg8hnz",
                                    },
                                    <context.deadlineExceededError>{},
                                ],
                            },
                        ],
                    },
                },
            },
        ],
    }
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
14:53:34 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x643438c6138>: 
    resource group "des-encrypt-mfwpss6hqvsd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="des-encrypt" in resourcegroup="des-encrypt-mfwpss6hqvsd" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234",
      "name": "a58b36b6-30fb-4aee-839d-12ae7b447234",
      "status": "Failed",
      "startTime": "2026-10-12T16:03:15.661006285Z",
      "endTime": "2026-10-12T16:27:21.295309747Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
            correlation ID: 0827ef63-2c1b-4cad-8799-bd4b41067d42
            client request ID: 2c486539-fc98-4112-bf4c-2bdff7d36160
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x643438c6138>: 
    resource group "des-encrypt-mfwpss6hqvsd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="des-encrypt" in resourcegroup="des-encrypt-mfwpss6hqvsd" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234",
      "name": "a58b36b6-30fb-4aee-839d-12ae7b447234",
      "status": "Failed",
      "startTime": "2026-10-12T16:03:15.661006285Z",
      "endTime": "2026-10-12T16:27:21.295309747Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a58b36b6-30fb-4aee-839d-12ae7b447234
            correlation ID: 0827ef63-2c1b-4cad-8799-bd4b41067d42
            client request ID: 2c486539-fc98-4112-bf4c-2bdff7d36160
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
14:53:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x351b8e4ae138>: 
    resource group "disabled-image-registry-pzs7vs68g9xt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-pzs7vs68g9xt" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13",
      "name": "f1e860d3-9723-4468-b91f-a96474515d13",
      "status": "Failed",
      "startTime": "2026-10-12T15:57:28.987630883Z",
      "endTime": "2026-10-12T16:21:31.7469599Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5a4g859cjm0hvqmru36k5u916p18 still exists (deletion dispatched at 2026-10-12T15:57:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
            correlation ID: eab286f4-49cc-48ea-8ec4-6c70b43ff1d0
            client request ID: 1a74a385-2e63-4deb-a488-b16ad26cc354
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x351b8e4ae138>: 
    resource group "disabled-image-registry-pzs7vs68g9xt": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-pzs7vs68g9xt" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13",
      "name": "f1e860d3-9723-4468-b91f-a96474515d13",
      "status": "Failed",
      "startTime": "2026-10-12T15:57:28.987630883Z",
      "endTime": "2026-10-12T16:21:31.7469599Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb5a4g859cjm0hvqmru36k5u916p18 still exists (deletion dispatched at 2026-10-12T15:57:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1e860d3-9723-4468-b91f-a96474515d13
            correlation ID: eab286f4-49cc-48ea-8ec4-6c70b43ff1d0
            client request ID: 1a74a385-2e63-4deb-a488-b16ad26cc354
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
14:53:34 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x1476512fa078>: 
    resource group "rg-cluster-nsg-subnet-reuse-xjg677xdgtps": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xjg677xdgtps" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "name": "d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:47.33491695Z",
      "endTime": "2026-10-12T16:22:51.509357112Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb58jqqdu8adgelm46n5haqq88idqc still exists (deletion dispatched at 2026-10-12T15:58:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
            correlation ID: f7d4d296-e895-4942-ace3-b59e68657889
            client request ID: 314b1f02-1204-4002-a941-604f43078239
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x1476512fa078>: 
    resource group "rg-cluster-nsg-subnet-reuse-xjg677xdgtps": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xjg677xdgtps" to finish deleting: GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "name": "d6de2ca4-02ff-4736-bb7c-5d1de1b40369",
      "status": "Failed",
      "startTime": "2026-10-12T15:58:47.33491695Z",
      "endTime": "2026-10-12T16:22:51.509357112Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdb58jqqdu8adgelm46n5haqq88idqc still exists (deletion dispatched at 2026-10-12T15:58:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5076096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6de2ca4-02ff-4736-bb7c-5d1de1b40369
            correlation ID: f7d4d296-e895-4942-ace3-b59e68657889
            client request ID: 314b1f02-1204-4002-a941-604f43078239
occurred
2h 24m 41s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f51faac
 westus3 e2e failed #7311 (open) 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
14:47:36 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xaa70fc27a80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-pcvlks6fdkqz" to finish creating: GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "name": "26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:23.241617081Z",
      "endTime": "2026-10-12T16:21:32.102166958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-q4ttl-4vcvt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-pcvlks6fdkqz--managed/ingress-cert-le-np-1-q4ttl-4vcvt-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-q4ttl-4vcvt-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: RetryableError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"RetryableError\",\n    \"message\": \"A retryable error occurred.\",\n    \"details\": [\n      {\n        \"code\": \"RetryableErrorDueToTooManyCalls\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX was used to perform too many calls within last 5 minutes. The number of ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
            correlation ID: bfc832f2-b15d-4a34-8c01-edecd304fc1a
            client request ID: cd0488d8-da8e-467d-8248-7320c5456b91
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xaa70fc27a80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-pcvlks6fdkqz" to finish creating: GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "name": "26ef0fd5-d97a-49a6-8fbf-0d954df5101e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:23.241617081Z",
      "endTime": "2026-10-12T16:21:32.102166958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-q4ttl-4vcvt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-q4ttl-4vcvt: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-pcvlks6fdkqz--managed/ingress-cert-le-np-1-q4ttl-4vcvt-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-pcvlks6fdkqz--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-q4ttl-4vcvt-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: RetryableError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"RetryableError\",\n    \"message\": \"A retryable error occurred.\",\n    \"details\": [\n      {\n        \"code\": \"RetryableErrorDueToTooManyCalls\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX was used to perform too many calls within last 5 minutes. The number of ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2636160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26ef0fd5-d97a-49a6-8fbf-0d954df5101e
            correlation ID: bfc832f2-b15d-4a34-8c01-edecd304fc1a
            client request ID: cd0488d8-da8e-467d-8248-7320c5456b91
2h 15m 48s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 18c49d4
 centralus e2e failed #7440 (open) 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift
14:42:30 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x2ec259084840>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-tv5qjjvw6mt5" to finish creating: GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16",
      "name": "639bbef1-ebbd-4367-896d-cec3c0263d16",
      "status": "Failed",
      "startTime": "2026-10-12T15:42:56.655438798Z",
      "endTime": "2026-10-12T15:43:02.11596193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
            correlation ID: 1ceff82d-31f8-4b03-ac69-9b0cd1c3b903
            client request ID: 24f2f909-e5ab-41c2-8601-8e5d6aea83f8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x2ec259084840>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-tv5qjjvw6mt5" to finish creating: GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16",
      "name": "639bbef1-ebbd-4367-896d-cec3c0263d16",
      "status": "Failed",
      "startTime": "2026-10-12T15:42:56.655438798Z",
      "endTime": "2026-10-12T15:43:02.11596193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8141056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/639bbef1-ebbd-4367-896d-cec3c0263d16
            correlation ID: 1ceff82d-31f8-4b03-ac69-9b0cd1c3b903
            client request ID: 24f2f909-e5ab-41c2-8601-8e5d6aea83f8
occurred
2h 12m 33s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2dd7b8b
 westus3 e2e failed #7397 (open) 1
e2e (1)
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
13:54:09 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x72d88398300>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2m7pgwml79qx": PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
            correlation ID: 0548989b-5739-4134-ae92-44e0b94372c4
            client request ID: eff3cdde-c6fa-4be6-bf0f-91fa006cb430
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x72d88398300>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2m7pgwml79qx": PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://rp.j4532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2m7pgwml79qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
            correlation ID: 0548989b-5739-4134-ae92-44e0b94372c4
            client request ID: eff3cdde-c6fa-4be6-bf0f-91fa006cb430
occurred
2h 28m 1s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 521d3fc
 centralus n/a passed #7376 (open) 0 n/a
2h 7m 2s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2ebef17 · merge 2ebef17
 centralus n/a passed #7466 (merged) 0 n/a
2h 31m 51s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 21c5097
 centralus alert, e2e failed ★#7162 (open) 2
e2e (1)
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36
12:58:13 UTC · failed at e2e · test SRE advances a control plane to an SRE-pinned version · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.005s.
cluster "sre-version-pin-l5rcnr" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.005s.
cluster "sre-version-pin-l5rcnr" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
12:58:13 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T14:00:36Z
  Ended:       2026-10-12T14:17:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4606592-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T14:00:36Z
  Ended:       2026-10-12T14:17:36Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4606592-svc", component="backend", name="csstatedump", region="centralus", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 22m 6s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head a7588ae
 centralus alert failed #6746 (open) 1
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
12:31:13 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:41:55Z
  Ended:       2026-10-12T13:54:52Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7788032-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 24m 15s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 3c74b09
 westus3 n/a passed #7473 (closed) 0 n/a
2h 29m 4s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1f96e5f
 westus3 e2e failed ★#7471 (open) 4
e2e (4)
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
12:06:11 UTC · failed at e2e · test Immutable cluster configuration rejects an immutable cluster DNS prefix PATCH without changing the shared cluster · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
12:06:11 UTC · failed at e2e · test Immutable node pool configuration rejects an immutable node pool OS disk size PATCH without changing the shared node pool · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
12:06:11 UTC · failed at e2e · test External auth issuer validation rejects an invalid external auth issuer URL without leaving a child or changing the shared cluster · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
12:06:11 UTC · failed at e2e · test Customer should get an existing HCP cluster and report a missing cluster through the RP · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-v4t896dszvwt": PUT https://rp.j3300864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-v4t896dszvwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
2h 30m 4s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 395ae23
 centralus alert failed #7467 (open) 1
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
11:48:16 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:04:34Z
  Ended:       2026-10-12T13:26:30Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2946432-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 45m 26s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head bf28540
 westus3 alert, e2e failed ★#7472 (open) 9
e2e (7)
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:52 UTC · failed at e2e · test Immutable node pool configuration rejects an immutable node pool OS disk size PATCH without changing the shared node pool · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:52 UTC · failed at e2e · test External auth issuer validation rejects an invalid external auth issuer URL without leaving a child or changing the shared cluster · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:52 UTC · failed at e2e · test Node pool create validation rejects a negative replica count without creating a node pool or changing the shared cluster · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:52 UTC · failed at e2e · test Customer should get an existing HCP cluster and report a missing cluster through the RP · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
11:44:52 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-mb49f2vf2mck/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-mb49f2vf2mck/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:52 UTC · failed at e2e · test Immutable cluster configuration rejects an immutable cluster DNS prefix PATCH without changing the shared cluster · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-vlx4fgdsmn4s": PUT https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-vlx4fgdsmn4s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
11:44:52 UTC · failed at e2e · test Engineering want metrics from HCP to be present in Azure Monitor Workspace · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x2f223f7aa0d8>: 
    resource group "hcp-metrics-lh6cn9z9mpvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-lh6cn9z9mpvb" to finish deleting: GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "name": "666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "status": "Failed",
      "startTime": "2026-10-12T13:00:52.055384192Z",
      "endTime": "2026-10-12T13:24:57.118445449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td8hkede6pik7afuli4rqulo2mdm0ib still exists (deletion dispatched at 2026-10-12T13:00:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
            correlation ID: 32f74568-8662-4b4e-8805-1aefdc453b8a
            client request ID: 470d44cf-4554-4545-aed1-856e45d54cdb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x2f223f7aa0d8>: 
    resource group "hcp-metrics-lh6cn9z9mpvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-lh6cn9z9mpvb" to finish deleting: GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "name": "666cf1a9-76c6-4e79-8657-ba56dbe93c28",
      "status": "Failed",
      "startTime": "2026-10-12T13:00:52.055384192Z",
      "endTime": "2026-10-12T13:24:57.118445449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td8hkede6pik7afuli4rqulo2mdm0ib still exists (deletion dispatched at 2026-10-12T13:00:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4879872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/666cf1a9-76c6-4e79-8657-ba56dbe93c28
            correlation ID: 32f74568-8662-4b4e-8805-1aefdc453b8a
            client request ID: 470d44cf-4554-4545-aed1-856e45d54cdb
occurred
alert (2)
alert [hcp] KubeDeploymentReplicasMismatch fired
11:44:52 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:44:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:44:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:45:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:45:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.152:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4879872/providers/microsoft.monitor/accounts/hcps-j4879872", namespace="ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9ncdp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8hkede6pik7afuli4rqulo2mdm0ib-ksm-hcp-metrics/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
alert [hcp] KubePodNotReady fired
11:44:52 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 20 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:30:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o", pod="azure-disk-csi-driver-controller-67b988dbd6-9pjgh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o/azure-disk-csi-driver-controller-67b988dbd6-9pjgh has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:25:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d", pod="multus-admission-controller-86cf4f5c69-mqbpc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d/multus-admission-controller-86cf4f5c69-mqbpc has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:25:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d", pod="multus-admission-controller-86cf4f5c69-jhk97", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d/multus-admission-controller-86cf4f5c69-jhk97 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o", pod="azure-disk-csi-driver-controller-67b988dbd6-b9n46", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o/azure-disk-csi-driver-controller-67b988dbd6-b9n46 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o", pod="azure-disk-csi-driver-controller-67b988dbd6-cg97m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hjpvhu8glv53d47p2282741ri22c-k5r0i5u0f1o9r0o/azure-disk-csi-driver-controller-67b988dbd6-cg97m has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-12T13:06:43Z
  Ended:       2026-10-12T13:25:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d", pod="multus-admission-controller-86cf4f5c69-kpf7b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hhrh2u47723p4jk6svvgmgqch9ah-a3w0n4n2x3d1l7d/multus-admission-controller-86cf4f5c69-kpf7b has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-12T13:13:45Z
  Ended:       2026-10-12T13:24:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i", pod="openshift-apiserver-6b4ff7ff95-9kn96", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i/openshift-apiserver-6b4ff7ff95-9kn96 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-7cvh9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-7cvh9 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-8mskw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-8mskw has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-j42hz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-j42hz has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-mkdz4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-mkdz4 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-12T13:15:42Z
  Ended:       2026-10-12T13:36:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-5xwvh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-5xwvh has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-npkzv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-npkzv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-m8dx4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-m8dx4 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-pnr6f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-pnr6f has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-gw6tj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-gw6tj has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-5x6tc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-5x6tc has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-6lcvg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-6lcvg has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:24:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i", pod="openshift-apiserver-6b4ff7ff95-mksfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hdubi1998hba1cbuuv9q41lduclv-j3t7o4c6b6y3x7i/openshift-apiserver-6b4ff7ff95-mksfs has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-12T13:15:43Z
  Ended:       2026-10-12T13:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4879872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j", pod="multus-admission-controller-66868c485c-bx6hr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8hir7bcqf7v5h045nvaaq6p25d2at-n3r7k8t0l9m8z2j/multus-admission-controller-66868c485c-bx6hr has been in a non-ready state for longer than 5 minutes.
2h 46m 1s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f3cccb6
 westus3 alert, e2e failed ⚠#7470 (open) 10
e2e (5)
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:03 UTC · failed at e2e · test Immutable cluster configuration rejects an immutable cluster DNS prefix PATCH without changing the shared cluster · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: DeadlineExceeded; detail code KeyVaultAccessForbidden; detail message Unable to access key vault resource '<url>' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated; provider Microsoft.RedHatOpenShift
11:44:03 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x14dd1e091060>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-kdz84x79jt45" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "name": "75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "status": "Failed",
      "startTime": "2026-10-12T12:59:06.888844222Z",
      "endTime": "2026-10-12T13:28:08.456404461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine des-encrypt-des-np-f8895-79vxf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine des-encrypt-des-np-f8895-gfvrv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine des-encrypt-des-np-f8895-79vxf: Failed: virtualmachine failed to create or update. err: failed to create or update resource des-encrypt-kdz84x79jt45--managed/des-encrypt-des-np-f8895-79vxf (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-kdz84x79jt45--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-f8895-79vxf\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: KeyVaultAccessForbidden\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"KeyVaultAccessForbidden\",\n    \"message\": \"Unable to access key vault resource 'https://cust-kv-knxkedtao2ry2.vault.azure.net/keys/des-encryption-key/3126b361c5834ce5ad1e97723eb63855' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
            correlation ID: bd8a92be-4962-42e7-8e34-639e73d20045
            client request ID: 6eda585f-7541-486b-8dad-d5cd134191ff
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x14dd1e091060>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-kdz84x79jt45" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "name": "75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "status": "Failed",
      "startTime": "2026-10-12T12:59:06.888844222Z",
      "endTime": "2026-10-12T13:28:08.456404461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine des-encrypt-des-np-f8895-79vxf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine des-encrypt-des-np-f8895-gfvrv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine des-encrypt-des-np-f8895-79vxf: Failed: virtualmachine failed to create or update. err: failed to create or update resource des-encrypt-kdz84x79jt45--managed/des-encrypt-des-np-f8895-79vxf (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-kdz84x79jt45--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-f8895-79vxf\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: KeyVaultAccessForbidden\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"KeyVaultAccessForbidden\",\n    \"message\": \"Unable to access key vault resource 'https://cust-kv-knxkedtao2ry2.vault.azure.net/keys/des-encryption-key/3126b361c5834ce5ad1e97723eb63855' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
            correlation ID: bd8a92be-4962-42e7-8e34-639e73d20045
            client request ID: 6eda585f-7541-486b-8dad-d5cd134191ff
occurred
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:03 UTC · failed at e2e · test Immutable node pool configuration rejects an immutable node pool OS disk size PATCH without changing the shared node pool · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "shared-read-only": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
11:44:03 UTC · failed at e2e · test Customer should get an existing HCP cluster and report a missing cluster through the RP · suite rp-api-compat-all/parallel
Show raw failure
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
provision shared read-only cluster: provision shared node pool: failed starting nodepool creation "shared-read-only" for cluster "shared-read-only" in resourcegroup="shared-ro-lr2d54k7v8w5": PUT https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shared-ro-lr2d54k7v8w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shared-read-only/nodePools/shared-read-only
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MultipleErrorsOccurred",
    "message": "2 errors occurred",
    "details": [
      {
        "code": "InvalidRequestContent",
        "message": "Too long: may not be more than 15 bytes",
        "target": "id"
      },
      {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"shared-read-only\": (must be a valid DNS RFC 1035 label)",
        "target": "id"
      }
    ]
  }
}
--------------------------------------------------------------------------------
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
11:44:03 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x329d818e85e0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-7hcwvl89hlds" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa",
      "name": "af485567-16d3-466b-8cef-6a0e9bce52fa",
      "status": "Failed",
      "startTime": "2026-10-12T12:42:57.954228784Z",
      "endTime": "2026-10-12T13:11:58.34288805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
            correlation ID: 70b5741c-ea80-4b9a-9e33-ac7a70af1ce7
            client request ID: f176988a-7ecb-4ecf-8063-4ee5d27bb712
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x329d818e85e0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-7hcwvl89hlds" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa",
      "name": "af485567-16d3-466b-8cef-6a0e9bce52fa",
      "status": "Failed",
      "startTime": "2026-10-12T12:42:57.954228784Z",
      "endTime": "2026-10-12T13:11:58.34288805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/af485567-16d3-466b-8cef-6a0e9bce52fa
            correlation ID: 70b5741c-ea80-4b9a-9e33-ac7a70af1ce7
            client request ID: f176988a-7ecb-4ecf-8063-4ee5d27bb712
occurred
alert (5)
alert [hcp] KubePodCrashLooping fired
11:44:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:08:11Z
  Ended:       2026-10-12T13:34:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4d015a5d-c4df-4754-bdd2-c1b853805f1f"
  Description: Pod ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:09:11Z
  Ended:       2026-10-12T13:33:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4d015a5d-c4df-4754-bdd2-c1b853805f1f"
  Description: Pod ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubeStatefulSetReplicasMismatch fired
11:44:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:06:14Z
  Ended:       2026-10-12T13:34:15Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5f8cm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:06:14Z
  Ended:       2026-10-12T13:29:13Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5f8cm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert [svc] BackendControllerQueueDepthHigh fired
11:44:03 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:00:26Z
  Ended:       2026-10-12T13:09:30Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5598976-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert [hcp] KubeDeploymentReplicasMismatch fired
11:44:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:20:16Z
  Ended:       2026-10-12T13:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5f8cm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:21:13Z
  Ended:       2026-10-12T13:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.236:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5598976/providers/microsoft.monitor/accounts/hcps-j5598976", namespace="ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5f8cm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td8fkj2kvk4mcasoetkg4c7kcrid62p-cilium-cl/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert [hcp] KubePodNotReady fired
11:44:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 119 time(s) (93 unknown, 26 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-phq7k", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-phq7k has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-p9svn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-p9svn has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-sn7pp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-sn7pp has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-g4rgv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-g4rgv has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-4c79f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-4c79f has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-5d87c549dd-xft98", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-5d87c549dd-xft98 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-8pxss", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-8pxss has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-wp75c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-wp75c has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-6d4b4c596d-kgm8k", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-6d4b4c596d-kgm8k has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-5d87c549dd-jfjg4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-5d87c549dd-jfjg4 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-ptd4w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-ptd4w has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-84sz2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-84sz2 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-j6427", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-j6427 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-n7mnp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-n7mnp has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-6d4b4c596d-bvqqj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-6d4b4c596d-bvqqj has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-jxwtv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-jxwtv has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-c8g6x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-c8g6x has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-5d87c549dd-nsvfd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-5d87c549dd-nsvfd has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-6d4b4c596d-n9gt9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-6d4b4c596d-n9gt9 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-d7rgr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-d7rgr has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-v2kpf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-v2kpf has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-wbdqf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-wbdqf has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-d52gv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-d52gv has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-6d4b4c596d-p5krx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-6d4b4c596d-p5krx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-qklrl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-qklrl has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-956mt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-956mt has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-757rl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-757rl has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-4b6hz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-4b6hz has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-zckfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-zckfs has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:26:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="kube-controller-manager-866b6df4c4-wxk72", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/kube-controller-manager-866b6df4c4-wxk72 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-12T13:03:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p", pod="azure-file-csi-driver-controller-5d87c549dd-pcdzb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fllch98do8t1mnl5upc3i6hqfnnp-d9e2s3z6q2u6v1p/azure-file-csi-driver-controller-5d87c549dd-pcdzb has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-12T13:04:13Z
  Ended:       2026-10-12T13:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-7kcvb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-7kcvb has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:24:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-fb7bn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-fb7bn has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-96x7z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-96x7z has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-p2wsf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-p2wsf has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-kkqwz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-kkqwz has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-md6p4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-md6p4 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-lrll2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-lrll2 has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-xmbjm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-xmbjm has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-phkhn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-phkhn has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-r9jwz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-r9jwz has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-kxwxf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-kxwxf has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-8p6bv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-8p6bv has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-7rq4z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-7rq4z has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-jc4df", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-jc4df has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:24:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-2c6lt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-2c6lt has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-7qsp9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-7qsp9 has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:23:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-s5nkm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-s5nkm has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-mq9bs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-mq9bs has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-12T13:05:11Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-dq8cp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-dq8cp has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-g49bf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-g49bf has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-lqt44", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-lqt44 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-2krkg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-2krkg has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-mwzbm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-mwzbm has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:24:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-rsdjb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-rsdjb has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-8cq7w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-8cq7w has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-p8kv5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-p8kv5 has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-twjkp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-twjkp has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-vncn4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-vncn4 has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-4b79p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-4b79p has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-8dv7s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-8dv7s has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-44qv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-44qv4 has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-p428l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-p428l has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-zhslf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-zhslf has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:24:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-72sgc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-72sgc has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-sv54r", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-sv54r has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-d9rsz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-d9rsz has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-br96b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-br96b has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-t8xgx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-t8xgx has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-8t44f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-8t44f has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-tf65g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-tf65g has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-vs6dq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-vs6dq has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-8krt7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-8krt7 has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-btj2q", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-btj2q has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-rf8gr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-rf8gr has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-cq958", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-cq958 has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-dzn28", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-dzn28 has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-km8hh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-km8hh has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-m6f6k", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-m6f6k has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-c5bb7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-c5bb7 has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-ts6d9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-ts6d9 has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-bgwqx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-bgwqx has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-g9kxt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-g9kxt has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-lpthv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-lpthv has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-fdl8s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-fdl8s has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-xs5wf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-xs5wf has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-75bdt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-75bdt has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:20:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v", pod="ignition-server-proxy-7ddb9b85d9-vxr64", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8fn7evvpbjsrtq4vi1j4snca0v678-o5y8w5x8p9c0y1v/ignition-server-proxy-7ddb9b85d9-vxr64 has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w", pod="azure-disk-csi-driver-controller-5b8cc5d6c8-6rgs4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8foj4rgg64afk1khume99dsthl10u-o7a9e0w4b8q9k1w/azure-disk-csi-driver-controller-5b8cc5d6c8-6rgs4 has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="control-plane-operator-59ffc49d8c-4svlv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/control-plane-operator-59ffc49d8c-4svlv has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:23:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-wtzg6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-wtzg6 has been in a non-ready state for longer than 5 minutes.
Firing 92:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-8vm5m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-8vm5m has been in a non-ready state for longer than 5 minutes.
Firing 93:
  State:       Resolved
  Started:     2026-10-12T13:05:12Z
  Ended:       2026-10-12T13:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5598976-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c", pod="cluster-api-59c88c8d8f-7n5w9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2td8g5jiacb863c3jitg24j7abmo6c7h-s4f2t4t5y0i3a4c/cluster-api-59c88c8d8f-7n5w9 has been in a non-ready state for longer than 5 minutes.
2h 16m 12s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5051cc7
 westus3 e2e failed #7373 (open) 1
e2e (1)
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
11:12:15 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-2dtwt4gvf9v7/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-2dtwt4gvf9v7/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
2h 38m 58s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 209c55b
 centralus alert, e2e failed #7468 (open) 3
e2e (1)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
11:01:58 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xde424e4090>: 
    resource group "admin-credential-lifecycle-test-7p8rt54bzxxl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-cm6t76" in resourcegroup="admin-credential-lifecycle-test-7p8rt54bzxxl" to finish deleting: GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad",
      "name": "d217aaa4-6911-41fe-a395-709b350055ad",
      "status": "Failed",
      "startTime": "2026-10-12T12:20:06.890936008Z",
      "endTime": "2026-10-12T12:44:10.847761888Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td7rmojjop6e5ni0mk3h0r2c95v0aqq still exists (deletion dispatched at 2026-10-12T12:20:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
            correlation ID: 5a65ad43-439a-49ac-a5b3-3ea36ef8c6fd
            client request ID: 3b679075-36d2-4af2-ad13-7fdd7af0d4bb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0xde424e4090>: 
    resource group "admin-credential-lifecycle-test-7p8rt54bzxxl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-cm6t76" in resourcegroup="admin-credential-lifecycle-test-7p8rt54bzxxl" to finish deleting: GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad",
      "name": "d217aaa4-6911-41fe-a395-709b350055ad",
      "status": "Failed",
      "startTime": "2026-10-12T12:20:06.890936008Z",
      "endTime": "2026-10-12T12:44:10.847761888Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2td7rmojjop6e5ni0mk3h0r2c95v0aqq still exists (deletion dispatched at 2026-10-12T12:20:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5767424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d217aaa4-6911-41fe-a395-709b350055ad
            correlation ID: 5a65ad43-439a-49ac-a5b3-3ea36ef8c6fd
            client request ID: 3b679075-36d2-4af2-ad13-7fdd7af0d4bb
occurred
alert (2)
alert [svc] BackendControllerQueueDepthHigh fired
11:01:58 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T12:20:39Z
  Ended:       2026-10-12T12:36:40Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5767424-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert [hcp] KubeDeploymentReplicasMismatch fired
11:01:58 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5767424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5767424/providers/microsoft.monitor/accounts/hcps-j5767424", namespace="ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5prtf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5767424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5767424/providers/microsoft.monitor/accounts/hcps-j5767424", namespace="ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5prtf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5767424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5767424/providers/microsoft.monitor/accounts/hcps-j5767424", namespace="ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5prtf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-12T13:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5767424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5767424/providers/microsoft.monitor/accounts/hcps-j5767424", namespace="ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5prtf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td7rmojjop6e5ni0mk3h0r2c95v0aqq-e8e5t4a2m8v7b5f/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
1h 59m 51s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2ebef17 · merge 2ebef17
 westus3 n/a passed #7466 (merged) 0 n/a
2h 47m 11s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 53e45f9
 centralus e2e failed ★#7383 (open) 1
e2e (1)
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36
11:00:15 UTC · failed at e2e · test SRE advances a control plane to an SRE-pinned version · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-shw9zg" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-shw9zg" did not complete the SRE-pinned version 4.21.36
Expected
    <string>: 4.21.36 (Partial)
to equal
    <string>: 4.21.36
2h 34m 38s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0c8ab0b
 centralus e2e failed #7457 (open) 1
e2e (1)
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
10:59:45 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-fbz6pthxmhdp/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-fbz6pthxmhdp/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
2h 31m 47s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4f30e2c · merge 4f30e2c
 centralus e2e failed ★#7420 (merged) 3
e2e (3)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-<id> to exist; provider Microsoft.RedHatOpenShift
10:14:00 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1bbf606b9160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-frt9kkhsnfm2" to finish creating: GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/75475c40-7fa1-4e8f-bf17-baaab7fc2f88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/75475c40-7fa1-4e8f-bf17-baaab7fc2f88",
      "name": "75475c40-7fa1-4e8f-bf17-baaab7fc2f88",
      "status": "Failed",
      "startTime": "2026-10-12T11:36:28.504164951Z",
      "endTime": "2026-10-12T12:05:35.482369546Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-qmq6h-h7gmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-frt9kkhsnfm2--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qmq6h-h7gmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-qmq6h-h7gmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-frt9kkhsnfm2--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qmq6h-h7gmx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/75475c40-7fa1-4e8f-bf17-baaab7fc2f88
            correlation ID: d89bc198-69ae-4651-acfe-b566b134eb9a
            client request ID: d8b88a68-6fe2-4293-891c-8ad5621c82b5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1bbf606b9160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-frt9kkhsnfm2" to finish creating: GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/75475c40-7fa1-4e8f-bf17-baaab7fc2f88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/75475c40-7fa1-4e8f-bf17-baaab7fc2f88",
      "name": "75475c40-7fa1-4e8f-bf17-baaab7fc2f88",
      "status": "Failed",
      "startTime": "2026-10-12T11:36:28.504164951Z",
      "endTime": "2026-10-12T12:05:35.482369546Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-qmq6h-h7gmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-frt9kkhsnfm2--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qmq6h-h7gmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-qmq6h-h7gmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-frt9kkhsnfm2--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qmq6h-h7gmx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/75475c40-7fa1-4e8f-bf17-baaab7fc2f88
            correlation ID: d89bc198-69ae-4651-acfe-b566b134eb9a
            client request ID: d8b88a68-6fe2-4293-891c-8ad5621c82b5
occurred
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-11T11:53:04
10:14:00 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.002s.
Expected success, but got an error:
    <*fmt.wrapError | 0x2582980c0060>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);
    let _endTime = datetime(2026-10-12T11:53:04.2658821Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-82f6cfwmh699'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-12T11:53:10.3021968Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1", ActivityId:"583de96c-d634-44b3-9661-ce011792b0cc", SubActivityId:"583de96c-d634-44b3-9661-ce011792b0cc", ActivityType:"GW.Http.CallContext", ParentActivityId:"583de96c-d634-44b3-9661-ce011792b0cc", ActivityStack:"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x258297402990>{
            errs: [
                <*fmt.wrapError | 0x2582980c0040>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x2582980c0020>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x2582961135c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x258296113580>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x258296cdc200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.002s.
Expected success, but got an error:
    <*fmt.wrapError | 0x2582980c0060>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);
    let _endTime = datetime(2026-10-12T11:53:04.2658821Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-82f6cfwmh699'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-12T11:53:10.3021968Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1", ActivityId:"583de96c-d634-44b3-9661-ce011792b0cc", SubActivityId:"583de96c-d634-44b3-9661-ce011792b0cc", ActivityType:"GW.Http.CallContext", ParentActivityId:"583de96c-d634-44b3-9661-ce011792b0cc", ActivityStack:"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x258297402990>{
            errs: [
                <*fmt.wrapError | 0x2582980c0040>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x2582980c0020>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x2582961135c0>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x258296113580>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-11T11:53:04.2658819Z);\nlet _endTime = datetime(2026-10-12T11:53:04.2658821Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-82f6cfwmh699'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-12T11:53:10.3021968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;48584352-556a-4a9a-886f-653d24baecf1\", ActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", SubActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"583de96c-d634-44b3-9661-ce011792b0cc\", ActivityStack:\"(Activity stack: CRID=KGC.execute;48584352-556a-4a9a-886f-653d24baecf1 ARID=583de96c-d634-44b3-9661-ce011792b0cc > GW.Http.CallContext/583de96c-d634-44b3-9661-ce011792b0cc)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x258296cdc200>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-<id> to exist; provider Microsoft.RedHatOpenShift
10:14:00 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x361e46553da0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dx8x9nb92ts2" to finish creating: GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/09be8631-a40d-4148-b3b2-3144216d4fe7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/09be8631-a40d-4148-b3b2-3144216d4fe7",
      "name": "09be8631-a40d-4148-b3b2-3144216d4fe7",
      "status": "Failed",
      "startTime": "2026-10-12T11:44:57.797440876Z",
      "endTime": "2026-10-12T12:14:05.478563828Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-48lvb-qrq6s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-dx8x9nb92ts2--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-48lvb-qrq6s to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-48lvb-qrq6s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-dx8x9nb92ts2--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-48lvb-qrq6s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/09be8631-a40d-4148-b3b2-3144216d4fe7
            correlation ID: 06eef5ee-18e5-459b-b472-73166e064f53
            client request ID: c3ba459a-88fd-41d3-aa0f-d4135fceec1f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x361e46553da0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-dx8x9nb92ts2" to finish creating: GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/09be8631-a40d-4148-b3b2-3144216d4fe7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/09be8631-a40d-4148-b3b2-3144216d4fe7",
      "name": "09be8631-a40d-4148-b3b2-3144216d4fe7",
      "status": "Failed",
      "startTime": "2026-10-12T11:44:57.797440876Z",
      "endTime": "2026-10-12T12:14:05.478563828Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-48lvb-qrq6s: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-dx8x9nb92ts2--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-48lvb-qrq6s to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-48lvb-qrq6s: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-dx8x9nb92ts2--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-48lvb-qrq6s to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4754816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/09be8631-a40d-4148-b3b2-3144216d4fe7
            correlation ID: 06eef5ee-18e5-459b-b472-73166e064f53
            client request ID: c3ba459a-88fd-41d3-aa0f-d4135fceec1f
occurred
2h 22m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5e5ec5c · merge 5e5ec5c
 centralus n/a passed #7458 (merged) 0 n/a
1h 3m 1s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head cb92dba
 westus3 alert, provision failed ★#7439 (open) 2
provision (1)
failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 13/15 context deadline exceeded
09:45:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/kubelet-ds (stamp=2) · suite templatize-pipeline
Show raw failure
time=2026-10-12T10:35:56.708Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-10-12T10:35:57.392Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 values=map[enabled:true]
time=2026-10-12T10:35:57.392Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-12T10:35:57.601Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 namespace=kubelet-ds
time=2026-10-12T10:35:57.906Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 dryRun=false
time=2026-10-12T10:35:57.907Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-12T10:35:58.041Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-12T10:40:59.966Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 13/15\ncontext deadline exceeded"
time=2026-10-12T10:40:59.967Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 13/15\ncontext deadline exceeded"
alert (1)
alert [svc] KubePodNotReady fired
09:45:06 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:37:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0004864-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-vk9xs", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-vk9xs has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:42:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0004864-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-64b5859f7f-5glfr", region="westus3", severity="warning"
  Description: Pod sessiongate/sessiongate-64b5859f7f-5glfr has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:42:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0004864-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-76c4b4c8bc-8gvkr", region="westus3", severity="warning"
  Description: Pod fleet/fleet-controller-76c4b4c8bc-8gvkr has been in a non-ready state for longer than 5 minutes.
1h 5m 16s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 21c5097
 westus3 alert, provision failed ★#7162 (open) 2
provision (1)
failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 13/15 context deadline exceeded
09:35:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/kubelet-ds (stamp=2) · suite templatize-pipeline
Show raw failure
time=2026-10-12T10:30:06.815Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-10-12T10:30:07.429Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 values=map[enabled:true]
time=2026-10-12T10:30:07.429Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-12T10:30:07.671Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 namespace=kubelet-ds
time=2026-10-12T10:30:08.147Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 dryRun=false
time=2026-10-12T10:30:08.147Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-12T10:30:08.275Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-10-12T10:35:10.285Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 13/15\ncontext deadline exceeded"
time=2026-10-12T10:35:10.286Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 13/15\ncontext deadline exceeded"
alert (1)
alert [svc] KubePodNotReady fired
09:35:22 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1804928-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14292482-vmss000002-dskt8", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-14292482-vmss000002-dskt8 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1804928-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14292482-vmss000000-472fv", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-14292482-vmss000000-472fv has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:31:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1804928-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dmqmp", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-dmqmp has been in a non-ready state for longer than 5 minutes.
1h 1m 25s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2ebef17 · merge 2ebef17
 westus3 provision failed ★#7466 (merged) 1
provision (1)
failed to roll out Helm release: resource Deployment/acrpull/acrpull not ready. status: InProgress, message: Available: 1/2 context deadline exceeded
09:26:31 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/acrpull · suite templatize-pipeline
Show raw failure
time=2026-10-12T10:17:07.734Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull description="Step acrpull\n Kind: Helm\n"
time=2026-10-12T10:17:08.348Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:b0cbba7044d5142f8545758e5a69b7b35e7f8c6af596bc972e164d2b345978a7 namespace:acrpull replicas:2 ttlRotationFraction:0.5]"
time=2026-10-12T10:17:08.348Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull
time=2026-10-12T10:17:08.589Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull namespace=acrpull
time=2026-10-12T10:17:08.838Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull dryRun=false
time=2026-10-12T10:17:08.838Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull
time=2026-10-12T10:17:08.976Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull
time=2026-10-12T10:22:16.134Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="resource Deployment/acrpull/acrpull not ready. status: InProgress, message: Available: 1/2\ncontext deadline exceeded"
time=2026-10-12T10:22:16.135Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/acrpull/acrpull not ready. status: InProgress, message: Available: 1/2\ncontext deadline exceeded"
2h 11m 22s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head ddfc51e
 centralus n/a passed #7421 (open) 0 n/a
2h 21m 42s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0e1c991
 centralus alert failed #7219 (open) 1
alert (1)
alert [svc] LeaderElectionLeaseStale fired
08:45:16 UTC · failed at alert · test [aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T10:44:16Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1015808-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.213:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1015808/providers/microsoft.monitor/accounts/services-j1015808", namespace="klusterlet-2td5olqiam9j0qubto977v59hbae6bo7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-m2h4p", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2td5olqiam9j0qubto977v59hbae6bo7 on cluster ci01-j1015808-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
2h 16m 32s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5aaf36c
 centralus alert, e2e failed #7463 (open) 5
e2e (2)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
08:35:17 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0x32753cabb000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lcmmtbwtg46t" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "name": "b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "status": "Failed",
      "startTime": "2026-10-12T09:39:05.150325854Z",
      "endTime": "2026-10-12T09:39:46.941270418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
            correlation ID: 84029b8a-bf49-4092-9e8c-07ae2562e0b2
            client request ID: 168cc3ec-3b47-4824-95c6-847f6de09edb
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0x32753cabb000>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lcmmtbwtg46t" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "name": "b663ffa3-8a99-44d9-a132-5e5b41ecbd2f",
      "status": "Failed",
      "startTime": "2026-10-12T09:39:05.150325854Z",
      "endTime": "2026-10-12T09:39:46.941270418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b663ffa3-8a99-44d9-a132-5e5b41ecbd2f
            correlation ID: 84029b8a-bf49-4092-9e8c-07ae2562e0b2
            client request ID: 168cc3ec-3b47-4824-95c6-847f6de09edb
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
08:35:17 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x3d2ba8a26f40>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-tmxkg7m275vm" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301",
      "name": "06ab8b07-36db-4738-9b5d-4d16092f8301",
      "status": "Failed",
      "startTime": "2026-10-12T09:26:40.450447729Z",
      "endTime": "2026-10-12T09:55:40.068785838Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
            correlation ID: c5b7b9a4-edc0-484e-9068-a103f50c2ab8
            client request ID: d0e69db4-dd9a-407e-93a4-3206c81ac5a6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x3d2ba8a26f40>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-tmxkg7m275vm" to finish creating: GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301",
      "name": "06ab8b07-36db-4738-9b5d-4d16092f8301",
      "status": "Failed",
      "startTime": "2026-10-12T09:26:40.450447729Z",
      "endTime": "2026-10-12T09:55:40.068785838Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9143424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/06ab8b07-36db-4738-9b5d-4d16092f8301
            correlation ID: c5b7b9a4-edc0-484e-9068-a103f50c2ab8
            client request ID: d0e69db4-dd9a-407e-93a4-3206c81ac5a6
occurred
alert (3)
alert [hcp] KubeStatefulSetReplicasMismatch fired
08:35:17 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T09:49:03Z
  Ended:       2026-10-12T10:13:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4hsb2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T09:49:03Z
  Ended:       2026-10-12T10:17:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4hsb2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert [hcp] KubePodCrashLooping fired
08:35:17 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T09:50:56Z
  Ended:       2026-10-12T10:17:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8f3cafcc-c004-4fb5-b689-5b2c6b0de186"
  Description: Pod ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-12T09:50:56Z
  Ended:       2026-10-12T10:17:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8f3cafcc-c004-4fb5-b689-5b2c6b0de186"
  Description: Pod ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubeDeploymentReplicasMismatch fired
08:35:17 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T10:03:02Z
  Ended:       2026-10-12T10:13:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9143424-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.60:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9143424/providers/microsoft.monitor/accounts/hcps-j9143424", namespace="ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4hsb2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td5jk909ajmhn3gj3ep0upql88758lp-v20261001/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
2h 19m 55s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head a7588ae
 centralus alert failed #6746 (open) 1
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
08:00:48 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T09:09:06Z
  Ended:       2026-10-12T09:25:06Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4826368-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 18m 42s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 19b4dbe
 centralus e2e failed #7237 (open) 1
e2e (1)
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
07:37:59 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-mpwdzx"
Unexpected error:
    <*fmt.wrapError | 0x149c10e309a0>: 
    DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x149c10e30940>{
            msg: "DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-mpwdzx"
Unexpected error:
    <*fmt.wrapError | 0x149c10e309a0>: 
    DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x149c10e30940>{
            msg: "DNS for agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-74hfg.apps.aro.v8f6d9e5u4q8w4o.16el.j9630464.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
2h 5m 55s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 10e4605 · merge 10e4605
 westus3 n/a passed #7465 (merged) 0 n/a
2h 15m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 10b61f3
 centralus e2e failed #7451 (open) 1
e2e (1)
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
06:55:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-s9x9xg"
Unexpected error:
    <*fmt.wrapError | 0x1d3dff18b020>: 
    DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1d3dff18afe0>{
            msg: "DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-s9x9xg"
Unexpected error:
    <*fmt.wrapError | 0x1d3dff18b020>: 
    DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1d3dff18afe0>{
            msg: "DNS for agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fmjrc.apps.aro.q1s9i9u1g0l4i3y.v4x2.j6720256.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
2h 22m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 4aee7aa
 centralus e2e failed #7440 (open) 2
e2e (2)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
06:48:01 UTC · failed at e2e · test ARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.1 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:160]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2462d4d4a5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5hsxwfczjwp5" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "name": "466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "status": "Failed",
      "startTime": "2026-10-12T07:57:35.126724285Z",
      "endTime": "2026-10-12T08:26:42.188209535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
            correlation ID: df9f6d3b-b689-4825-a253-151f9584e7e4
            client request ID: 5fd0c7fe-22bf-4239-aeb7-b1d20c9c430c
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:160]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x2462d4d4a5a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-5hsxwfczjwp5" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "name": "466d7615-8a3d-4982-a2e2-4e4a80210d6e",
      "status": "Failed",
      "startTime": "2026-10-12T07:57:35.126724285Z",
      "endTime": "2026-10-12T08:26:42.188209535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-5g5xl-j5pmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5hsxwfczjwp5--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-5g5xl-j5pmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/466d7615-8a3d-4982-a2e2-4e4a80210d6e
            correlation ID: df9f6d3b-b689-4825-a253-151f9584e7e4
            client request ID: 5fd0c7fe-22bf-4239-aeb7-b1d20c9c430c
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
06:48:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x23adf9021e40>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-92gd76g6d97g" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
            correlation ID: 349f9734-994e-46db-888e-e1caf77e8fb4
            client request ID: 46549edd-5c88-4d3d-b17b-df9c20ad793d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x23adf9021e40>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-92gd76g6d97g" to finish creating: GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6616960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/32d45c3f-af20-4089-8a66-e8ffd728ab43
            correlation ID: 349f9734-994e-46db-888e-e1caf77e8fb4
            client request ID: 46549edd-5c88-4d3d-b17b-df9c20ad793d
occurred
2h 19m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5e5ec5c · merge 5e5ec5c
 westus3 alert failed #7458 (merged) 1
alert (1)
alert [svc] LeaderElectionLeaseStale fired
06:17:57 UTC · failed at alert · test [aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T08:19:01Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j8626816-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626816/providers/microsoft.monitor/accounts/services-j8626816", namespace="klusterlet-2td3l5vvknp9tr2ukktok1ca6hp11kor", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8hz5j", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2td3l5vvknp9tr2ukktok1ca6hp11kor on cluster ci01-j8626816-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
2h 30m 17s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c154328 · merge c154328
 westus3 alert failed #7424 (merged) 2
alert (2)
alert [svc] LeaderElectionLeaseStale fired
06:17:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T08:22:04Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j7802496-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7802496/providers/microsoft.monitor/accounts/services-j7802496", namespace="klusterlet-2td3ln668dv56op2785f7bjdpr831gpj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qmpf7", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2td3ln668dv56op2785f7bjdpr831gpj on cluster ci01-j7802496-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert [hcp] KubeDeploymentReplicasMismatch fired
06:17:02 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-12T08:05:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7802496-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7802496/providers/microsoft.monitor/accounts/hcps-j7802496", namespace="ocm-arohcpci01-2td3lsap1436jgm959cn57jh1risiuj3-f9r1b8n1e1c4l7y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qmpf7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td3lsap1436jgm959cn57jh1risiuj3-f9r1b8n1e1c4l7y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-12T08:12:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7802496-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7802496/providers/microsoft.monitor/accounts/hcps-j7802496", namespace="ocm-arohcpci01-2td3lsap1436jgm959cn57jh1risiuj3-f9r1b8n1e1c4l7y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qmpf7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2td3lsap1436jgm959cn57jh1risiuj3-f9r1b8n1e1c4l7y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
2h 7m 39s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c696b67
 centralus n/a passed #7097 (open) 0 n/a
2h 35m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 601ab01
 centralus alert failed #7430 (open) 1
alert (1)
alert [hcp] KubePodNotReady fired
05:40:39 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 13 time(s) (1 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T06:50:18Z
  Ended:       2026-10-12T07:11:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1507200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2td30fa9ubf174gpmmj5sp08qe3vi0pk-z3e0o7o8p5c7d6x", pod="ovnkube-control-plane-758b9b4b46-kgsrv", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2td30fa9ubf174gpmmj5sp08qe3vi0pk-z3e0o7o8p5c7d6x/ovnkube-control-plane-758b9b4b46-kgsrv has been in a non-ready state for longer than 5 minutes.
2h 10m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head cb92dba
 centralus e2e failed #7439 (open) 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
02:39:47 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc252182d240>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c847fqslv6ss" to finish creating: GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1",
      "name": "de9482bc-af9d-45d5-a419-3811860ecab1",
      "status": "Failed",
      "startTime": "2026-10-12T03:33:37.063419631Z",
      "endTime": "2026-10-12T04:02:41.894551349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
            correlation ID: 5048e7cb-ed4b-40b9-bfb8-b714b05f3bec
            client request ID: a436b4d6-2bc5-40f5-a8b1-84635292bdaa
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc252182d240>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c847fqslv6ss" to finish creating: GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1",
      "name": "de9482bc-af9d-45d5-a419-3811860ecab1",
      "status": "Failed",
      "startTime": "2026-10-12T03:33:37.063419631Z",
      "endTime": "2026-10-12T04:02:41.894551349Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c847fqslv6ss--managed/providers/Microsoft.Compute/virtualMachines/a2d1d2v0a9x4h6c-np-delete-main-hht52-lmtb9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1330688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/de9482bc-af9d-45d5-a419-3811860ecab1
            correlation ID: 5048e7cb-ed4b-40b9-bfb8-b714b05f3bec
            client request ID: a436b4d6-2bc5-40f5-a8b1-84635292bdaa
2h 13m 11s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c154328 · merge c154328
 westus3 e2e failed ★#7424 (merged) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
01:46:36 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x11fa8d633b80>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dmqcn9" in resourcegroup="rg-np-noedge-dmqcn9-8bfmwp2gm92p" to finish creating: GET https://rp.j7832576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5250eaac-747d-49c2-9ef4-8bb8ba7e096f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5250eaac-747d-49c2-9ef4-8bb8ba7e096f",
      "name": "5250eaac-747d-49c2-9ef4-8bb8ba7e096f",
      "status": "Failed",
      "startTime": "2026-10-12T03:07:24.415643123Z",
      "endTime": "2026-10-12T03:36:29.188004519Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx to exist\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx to exist\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7832576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5250eaac-747d-49c2-9ef4-8bb8ba7e096f
            correlation ID: 72fd1a70-e0c7-48fa-aa70-d0f6edab139d
            client request ID: 04d73669-7402-48f6-a103-fc2cac633166
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x11fa8d633b80>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dmqcn9" in resourcegroup="rg-np-noedge-dmqcn9-8bfmwp2gm92p" to finish creating: GET https://rp.j7832576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5250eaac-747d-49c2-9ef4-8bb8ba7e096f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5250eaac-747d-49c2-9ef4-8bb8ba7e096f",
      "name": "5250eaac-747d-49c2-9ef4-8bb8ba7e096f",
      "status": "Failed",
      "startTime": "2026-10-12T03:07:24.415643123Z",
      "endTime": "2026-10-12T03:36:29.188004519Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx to exist\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-7zfxx to exist\nMachine m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-dmqcn9-8bfmwp2gm92p-np-ne-dmqcn9--managed/providers/Microsoft.Compute/virtualMachines/m0u0w0c5m8f3v2m-npnoedge-4-20-zjc6m-r229g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7832576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5250eaac-747d-49c2-9ef4-8bb8ba7e096f
            correlation ID: 72fd1a70-e0c7-48fa-aa70-d0f6edab139d
            client request ID: 04d73669-7402-48f6-a103-fc2cac633166
occurred

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 43m 37s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth n/a passed n/a 0 n/a
2h 35m 37s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2 n/a passed n/a 0 n/a
28m 52s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
2h 57m 17s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westeurope n/a passed n/a 0 n/a
2h 46m 8s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
13:01:57 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-q7jjp7" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x298b49702ce0>: 
    failed to create HCP cluster cluster-zstream-4-21-q7jjp7: failed waiting for cluster="cluster-zstream-4-21-q7jjp7" in resourcegroup="rg-zstream-upgrade-4-21-jd6h7xk8btv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "name": "c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "status": "Failed",
      "startTime": "2026-10-12T14:02:09.836072198Z",
      "endTime": "2026-10-12T14:31:10.061357256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, packageserver, cluster-network-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
            correlation ID: 414509d7-91c9-4911-b025-fd41c0e067b1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-q7jjp7" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x298b49702ce0>: 
    failed to create HCP cluster cluster-zstream-4-21-q7jjp7: failed waiting for cluster="cluster-zstream-4-21-q7jjp7" in resourcegroup="rg-zstream-upgrade-4-21-jd6h7xk8btv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "name": "c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
      "status": "Failed",
      "startTime": "2026-10-12T14:02:09.836072198Z",
      "endTime": "2026-10-12T14:31:10.061357256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, packageserver, cluster-network-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
            correlation ID: 414509d7-91c9-4911-b025-fd41c0e067b1
            client request ID: unavailable
occurred
2h 30m 59s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 switzerlandnorth n/a passed n/a 0 n/a
2h 27m 30s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 canadacentral n/a passed n/a 0 n/a
2h 44m 41s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 centralindia n/a passed n/a 0 n/a
2h 44m 18s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a
2h 36m 17s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 switzerlandnorth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
06:42:34 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ptqdgd" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x21d67b55ea20>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-ptqdgd: failed waiting for cluster="cp-ystream-upgrade-5-0-ptqdgd" in resourcegroup="rg-cp-ystream-upgrade-5-0-dznstqd64m54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044",
      "name": "9d14066e-1629-48f8-aa68-f556c4fd7044",
      "status": "Failed",
      "startTime": "2026-10-12T07:51:24.545231929Z",
      "endTime": "2026-10-12T08:20:29.935309867Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
            correlation ID: 7a60ad0c-37f6-45ef-98dd-efdfad687d6c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ptqdgd" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x21d67b55ea20>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-ptqdgd: failed waiting for cluster="cp-ystream-upgrade-5-0-ptqdgd" in resourcegroup="rg-cp-ystream-upgrade-5-0-dznstqd64m54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044",
      "name": "9d14066e-1629-48f8-aa68-f556c4fd7044",
      "status": "Failed",
      "startTime": "2026-10-12T07:51:24.545231929Z",
      "endTime": "2026-10-12T08:20:29.935309867Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
            correlation ID: 7a60ad0c-37f6-45ef-98dd-efdfad687d6c
            client request ID: unavailable
occurred
2h 23m 33s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a
2h 22m 15s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a
14m 51s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
2h 23m 48s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a
2h 20m 23s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
00:58:22 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc79feec4060>: 
    resource group "np-autoscaling-xtg8j7kg4fwz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-xtg8j7kg4fwz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "name": "6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "status": "Failed",
      "startTime": "2026-10-12T02:47:19.509292457Z",
      "endTime": "2026-10-12T03:11:19.252244159Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc79feec4060>: 
    resource group "np-autoscaling-xtg8j7kg4fwz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-xtg8j7kg4fwz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "name": "6af3ef7c-4974-4b33-bebb-ed53f86cb854",
      "status": "Failed",
      "startTime": "2026-10-12T02:47:19.509292457Z",
      "endTime": "2026-10-12T03:11:19.252244159Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 31m 36s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 2
e2e (2)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist; provider Microsoft.RedHatOpenShift
00:05:13 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x38da2adfeb00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-kbw6v2d2bhz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095",
      "name": "4899c274-4471-4ab9-871b-22368aa84095",
      "status": "Failed",
      "startTime": "2026-10-12T00:35:15.167750325Z",
      "endTime": "2026-10-12T01:04:20.403308122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-fk5wl-zdgdp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-fk5wl-zdgdp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
            correlation ID: 79b16636-2655-4af8-9ac0-f8e4eda0ad35
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x38da2adfeb00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-kbw6v2d2bhz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095",
      "name": "4899c274-4471-4ab9-871b-22368aa84095",
      "status": "Failed",
      "startTime": "2026-10-12T00:35:15.167750325Z",
      "endTime": "2026-10-12T01:04:20.403308122Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-fk5wl-zdgdp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-fk5wl-zdgdp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
            correlation ID: 79b16636-2655-4af8-9ac0-f8e4eda0ad35
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
00:05:13 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x151c97ead140>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-p86tvztrm7hd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "name": "01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "status": "Failed",
      "startTime": "2026-10-12T01:36:23.272806003Z",
      "endTime": "2026-10-12T02:05:32.513113167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
            correlation ID: e3c8027d-936a-4cfa-802c-eac5497d3369
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x151c97ead140>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-p86tvztrm7hd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "name": "01aaf869-754a-4412-a8d0-48b576bc7f7b",
      "status": "Failed",
      "startTime": "2026-10-12T01:36:23.272806003Z",
      "endTime": "2026-10-12T02:05:32.513113167Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
            correlation ID: e3c8027d-936a-4cfa-802c-eac5497d3369
            client request ID: unavailable
occurred

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 57m 52s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a